FD 096 351 



DOCOHEHT BESDHB 



TB 003 967 



AOTHOB 

I!#STITUTION 

SPONS AGENCY 

.PCS D^TS ' 
MOTE 



EDPS PRICE 
DESTRIPTOKS 



Goodwin, wil'liani L. 

The- Laboratory of 'Educational Research (LEE) 
1966-1973. 

Colorado Univ., Boulder. Lab. of Educational 
Research. 

National Center for Educational Research and 

Development (DHEM/OE) , Was\hington, D.C. 

73 

84p.; For related doousents^^ See TM 003 968-973 . 

/ f!F-$0.75 HC-$4.20 PLOS POSTAGE 

♦Doctoral Prograas; *Educational Research; ♦Graduate 
Students; Models; ♦Program Evaluation; s*Suaiiative 
. P valuation; Training laboratories; Traim|ig, 
Tec^hniques 




ABSTRACT . 

The first ejection of this document presents an 
historical overview of the Labor<atory of Educational Research (IfER) 
Program of the University Colorado during the years 1966 to 1973. 
This section describes the con^et^tion and objectives of the LBB 
Program, its inpleBentationV thi mature of its faculty and students ^ 
(including the eBploynenp^patt*;rn of those students who have 
graduated fton the progrwii) , and its funding support levels. Section 
2 describes the LEI? Training Model and discusses staff, university 
support, student quality and requireaents, a detailed description of 
the prograafs curriculuBt and various changes which, have been oade in 
*-he prograa. Section 3 is a suaaative evaluation of the LER Prograa. 
It discusses ^he effects of the prograa o!\ the University of 
Colorado, its faculty and its situdents, effects on LER students, 
overall effect^ of the LER training prograa* and LEB's stre'ngths, 
weaknesses, and future. The questionnaire which was sent to the 
program's students for the purpose of evaluating LER and the 
inter view form which was used in questioning staff axe included as an 
appendix. The report also contains a number of tables which suaaarize 
the data collected during the evaluation. (Author/SE) 
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INTRODUCTION 

This document contains three major components. The first section is a 
historical overview of the LER program over the initial seven years of its 
existence. The second section details the major elements in the LER training 
model , with several near-suniiiative opinions on the adequacy of the elements. 
The final section contains a summary summative evaluation of the LER program. - 

Data sources for this document were twofold. First, information existing In 
the archives of LER, particularly several comprehensive evaluations conductetl in 

* 

past years, formed an exceptional base of data pertinent for this report. 
Second, new information was generated by soliciting vital opinions of all past 
and present LER students (the Questionnaire utilized Is included as App^dlx A) 
and by personal interviews with the two principal staff me*nbers (the sets of 
questions used are included as Appendix B). Some furtFer elaboration of the 
procedures used with these new data sources, and the reasons for the omission of 
other potential data sources, is in order. 

Procedures used In securing the opinions and vitae of LER students were 
standard in most regards, given this questionnaire-weary world. The question- 
naire was mailed to potential respondents. It was long, and a series of follow- 
up requests was necessary^ In all, 35 questionnaires were returned out of a 
possible 39, or 90 percent. Of the four non-respondents, two had left the 
program after less than one year's work — one of these returned the questlon- 
naire stating his willingness to respond, but also indicating feeling "out-of- 
it" in terms of most of the questions. Of the remaining two non-respondents, 
one sent in a vita, but indicated insufficient time to undertake the question- 
naire, while the final non-respondent was not heard from at all. (In passing, 
it should be noted that several unsolicited conments about instrument design 
and the length of the questionnaire were received in all, a knowledgeable but 
salty set of barbs.) 



The procedure used »to interview the principal staff of LER, Ken Hopkins and 
Gene Glass, initially involved the generation of 38 questions, some of them 
containing muUipl.e sub-questions, as given in Appendix B. (Both faculty also 
responded to selected items on the written questionnaire used for students.) 
The 38 questions were derived, from previous instruments used in LEK evaluations, 
from cownents and responses of students to the present questionnaire, and from 
this writer's personal knowledge of the LER program. To the extent possible, 
the 38 items were paired and then one question from each pair was assigned, 
randomly, to either Question Set A or Question Set B. Ken Hopkins was inter- 
viewed alone and responded to Question Set A; then Gene Glass responded indepen- 
dently to Question Set B. Responses were recorded in written form by the inter- 
viewer. Then, with both men present. Ken Hopkins responded briefly to questions 
earlier answered by Gene Glass, and vice-versa. When responses to the same 
questij^n were disparate, clarification and eleboration by the two men ensued. 
The entire session of interviews required six hours. 

The reader might legitimately ask why other new data sources were not 
included in compiling this report. Given limited resources with which to 
support this venture, a conscious decision was made to focus on the past and 
present fellows and the two principal LER faculty as they seemed to represent 
the best information sources bearing on both the history of the LER and the 
components of the LER training model. Further, LER students were reasoned to 
be the most visible and viable product of LER -- and their yitae seemed likely 
to contain, in large part, the most appropriate bits of evidence bearing on 
summative judgments about the worth of the LER training program. Thought was 
given to sampling present employers of LER graduates, to sampling School of 
'^''Education faculty opinion on LER, to content-analyzing and judging the research 
design quality of pre- and post-LER, theses by School of Education doctoral 
students, etc. These possibilities, however, were rejected either because of 



cost considerations or because of considerable certainty from previous evalua- 
tions, informal connujni cation, and informal analysis that data thus generated 
would be favorable to LER, or for both reasons. The reflections and opinions of 
past students and the LER faculty, and the records of student accomplishment via 
their vitae and the archives, were considered to be the most appropriate data 
bearing on the principal elements of this "report. 

A final introductory note is in order. This writer, through general long- 
term association with the students and faculty of LER, is hardly as objective, 
unbiased, and detached as some would hope considering the topical content of 
this document. Therefore, I state openly my high regard for the LER training 
program and its mentors, and for the students that have been nurtured under it. 
At the same time, I have endeavored to reporn accurately and to interpret fairly 
data I have encountered. This data, or other data, is open for interpretation 
by other interested parties. Like the^radio and television stations, I welcome 
the opposing views of responsible parties — any such views forthcoming in non- 
profane written form will be cheerfully appended, once received, to this 
document. 



I. HISTORICAL OVERVIEW ^ 

In time span, the history of the Laboratory of Educational Research Is 
short. In terms of events and impacts however, the history of LER appears 
substantial. " - . 

Eijcellent progress reports and evaluations have been made of LER since Its, 
conception and have previously been submitted to Washington and, so, are a matter 
of record — no attempt will be made to improve on those extensive documents. 
Rather, emphasis here will be on extracting key elements and reporting, in over- 
view form, the event history of Ler. 

A. The Improbabl e Conception 

Turn the clock back 10 years and examine the life and t\mes and research 
emphasis of the University of Colorado School of Education in the early I960's. 
Any sane observer asked then to predict the likelihood that a strong doctoral 
research training program would exist by the end of the decade would almost 
certainly have considered it highly improbable. The few courses offered by 
Education in the area were elementary and intermediate statistical methods, 
advanced educational measurement, evaluation of school systems and programs, and 
methods of educational research. In most of these courses, revision, if not 
wholesale revamping, was badly needed. Further, research was not a principal 
focus of the School of Education-, and virtually none of the faculty "spoke 
computer." This is not meant es an indictment of the School of Education then, 
but is presented only to illustrate Colorado's posture In this regard (a 
posture quite typical for education schools, colleges, and departments nation^ 
wide) and to highlight the seemingly fallow, if not sterile, environment for 
what was to follow. 
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A key event precipitating the development of LER was a recomnendation by 
the Education faculty's Committee on Research, Me^suremeut, and Statistics for 
the 1964-65 Academic Year. The Coirmittee encouraged the hiring of a staff memfaer 
wi;th specialization in educational research, measurement, and statistics. As a 
result. Dr. Ken Hopkins was hired and conmenced his work at the University of 
Colorado in September. 1965. 

IXjring his initial- year. Dr. Hopkins was instrumental in the development of 
tv/o companion proposals one for research program development and one for f 
support of research trainees. Two grants, subsequently awarded by the U.S. 
Office of Educations, allowed the establishment of the Laboratory of Educational 
Research in September, 1966. The principal objective of LER, and related sub- 
objectives, are quoted from the December, 1965, grant request: 

The basic objective of the program, then, is to attract, train, 
and graduate persons with competence in, and dedication to, 
educational research that will enable them to stimulate and 
conduct research studies focused on important aspects of educa- 
tion. The more immediate objectives of the proposal are the 
following; 

1. To develop and strengthen the specialized staffs in 
statistics, measurement, research design, and computer 
applications. 

2. To develop the curricular capability in order that the 
necessary academic and professional experience for 
specialization in educational research is provided. 

3. To provide the necessary relevant mat^als and equip- 
ment required for a comprehensive training program in 
educational research. 

\ 

B. The Ensuing Implementation 

Implementation might best be considered relative to each of the objectives 
quoted above. To a large extent, the "basic objective" has been maintained over 
the intervening time period. That is, new waves of research traine^%/f^3ve been 
launched at regular annual intervals. Most have been supported with fellowships 
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.awarded under the training grant, whi'le others have had support from elsewhere 
(e.g., NDEA) or have been self -supporting. . 

The three "immediate objectives" stated above have each led to substantial 
dch-fvements. Their very nature, however, kdeps them from being "permanently 
achieved," particularly as the LER has experienced a continuing growth in 

« 

responsibilities and user-trainee expectations. An examination of each immediate 
objective will make clear the acQompI fshments as well as the evaluation of new 
needs in the s^me areas. 

Staff development (Immediate Objective #1) primarily took the form of adding 
Or. Oene Glass to the LER facultv'in June, 1967. Dr. Glass and Dr. l4opkins have 
provided the cont.nuing core nucleus of the L£R faculty and have served as 
advisors for the bulk of the LER graduates. Other School of Education staff have 
also played supportive roles for LER: » 

(1) Dr. Hart)ld Anderson served as a prime force In Initiating and 
marshalling faculty sentiment for hiring a professor with research 
methodology as a specialization and an organization "1 Ike LER; he has 
also assisted in the development and teaching of several courses In 
the program area. 

(2) Or. Blaine Worthen joined the School of Education faculty In August, 
^ * .1969, and has provided several Internship opportunities for the LER 

students, particularly in connection with the AERA.Task Force Project 
that he directed. He also Increased LER Interest in the development 
of educational evaluation as a field and in training students to 
conduct educational evaluations. Dr. Worthen left the University In 
the summer of 1973 to assume a new post as Director of Research and 
Evaluation for the Northwest Regional Educational Laboratory. 

(3) Dr. William Goodwin joined th6 School of tducatlon faculty In August, 
1970, and has provided some internship opportunities for LER students, 
primarily on evaluations. He directed the Evaluation and Development 

^ Design Grant that terminated in December, 1970, and that proposed an 
exren^ ion/elaboration of the LER training model. 

Other School of Education staff, such as Dr. Ron Andersdn, have helped LER by 

teaching program area courses. Further, faculty from, other disciplines in the 

^University have offered highly relevant courses for LER stMdents, have served on 

their theses committees, etc. There Is also Veason to anticipate considerable 



support for the LER program froiTDr. Paul Bradley, who joined the School of 
Education faculty in August, 1973, after serving three years in the Center for 
the Study of Evaluation. (Dr. Worthen's departure* left a, position in research, 
sta^stics, and measurement open.) The increased distinction and niaturity of the 
LER faculty is obvious from their present accelerated growth of nwstaches and 
beards. "Despite this fact and the foregoing discussion, it would be misleading 
to imply that staffing no longer represents a problem for LER. Demand — for 
courses in the program area, for consultative assistance, and for individual 
assistance for students — has far outstripped the faculty personnel resources 
available. The 'criticality of this need has been accentuated by Dr. Glass's 
limited availability during his 1973-74 University Creative Fellowship Leave of 
Absence. Continuing attention will necessarily be devoted to this peed... in 
short, this first objective is, in some respects, still immediate. 

Curricular- capability (Inmediate Objective #2) was dramatically enhanced In 
the initial :.!;ar5 after LER creation. The Final Report for the Program Develop- 
ment Grant (Grant No. 0EG-8-8-961860-4003(058) , Project No. 6-1860) dated 
November, 196r, contains the pertinent summary reproduced below: 

[The grantn made possible the addition of several needed courses 
and experiences for research specialization in education: 

Educ. 604 - Experimental Design and Analysis I; 

Educ. 605 - Experimental Design and Analysis 11; 

Educ. 608 - Internship in Educational Research I; 

Educ. 609 - Internship in Educational Research II; 

Educ. 610 - Internship in Educational Research III; 

Educ. 611 - Internship in Educatiqnal Research IV; 

Educ. 695 - Seminar in Research Methodology. 

In addition, two other courses were increased from two to three 
credit hours to allow greater depth and breadth of coverage: 

*Educ. 480 - Elementary Statistical Methods; 

Educ. 511 - Advanced Educational Measurement and Evaluation. 

Two oth$r courses were completely updated and revamped: 

Educ. 591 - Evaluation of School Systems and Programs; 
Educ. 600 - Methods of Educational Research. 
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The LER trainees were also instrumental in creating the need 
for an additional course in the Computing Science Department: 

Comp. Sci. 451 -Computers in Behavioral Science 
Even with this new capability, continuing attention has necessarily been given to 
articulation between courses. and tfo modification of content (e.g., including 
computer applications in Educ- 480). \ 

Materials and equipment (Immediate Objective #3) were likewise In substantial 
evidence by the date of the final report for the program development grant. A 
library of over 1,100 volumes of professional periodicals and reference, books 

I 

had been assembled, a statistics laboratory with six automatic desk calculators 
had been established, and faculty, student, and secretarial stations had been 
equipped. Further^ the University had allocated 1,000 square feet of space In 
the basement of the McKenna Building for LER. This space was divided Into the 
statistical laboratory, two faculty offices, a desk and filing cabinet for each 
trainee, a secretarial area, and a seminar-consultation area. Minor alterations, 
primarily improvements, occurred ir^ the following year. 

In January, 1970, LER was moved to its present location, the Education 
'Annex. This is a converted brick house of considerable character and charm, 
appi^ximately 250 yards from Hell ems Annex that houses the main portion of the 
School of Education. In this facility, approximately 2,000 square feet are 
reserved for LER use. Four faculty and three secretarial offices are available 
as well as a large room (the "bull-pen") and two smaller rooms for trainee 
desks, filtng cabinets, and some desk calculators. A small conference room is 
also utilized. Another small room off the main secretarial station contains 
a temperamental, sensitive copying machine, other desk "calculators, and IBM 
kej(punch, and the slowly expanding library. The current facility of LER, 
despite the non-rhythmic pounding, of an antiquated heating system with two 
settings^ (off ond roasting) and the surprises and uncertainties of a phantom 
janitojflal service, has served LER wel 1 , 



In summary, considerable progress has been made on all three immediate 
objectives, although continuing attention to them has been necessary, particularly 
for the first (staff personnel) and second (curriculum content). The basic 
objective, with its implications of repeatedly training new waves of students, 
remains in the forefront. Major historical events related to LER, most of them 

* 

already discussed, are presented on the timeline in Figure 1. 

C. Fellows and Fellowees * 

Of critical and obvious importance in any historical review of LER are the 
students who have undergone and been part of the instructional program. In 
- Table 1, a listing of studentVwho have successfully completed the program, ,and 
those currently active in LER, is given. Not listed are the relatively few 
students who did not complete the LER program. ^ 

As indicated in the table, 25 students have received the Ph.D. via the LER 
program. Nearly all these students had substantial interaction, during their 
LER years, with Drs. Glass and Hopkins. In J:erms of thesis advising. Dr. Glass 
. advised about 50 percent of the LER completers, Dr. Hopkins abou^ 30 percent, 
while other School of Education staff served as thesis chairmen for 20 percent 
of the students. In addition to the 25 Ph.D. recipients, two other students 
have completed the coursework and left LER, but have not completed the disserta-/ 
tion (both are working at National Assessment in Denver). The remaining nine 
students are active students in LER, as of the Fall semester, 1973. 

The column in Table 1 identifying the number of positions or employers bf 
former LER students is difficult to interpret given the relatively brief ti»ie 
that the students have been in the job market. In general, it would seem that 
the LER completers represent a moderately place-mobile group, but not excessively 
so. Only one LER graduate has moved three times since completing the program. 
In terms of type-of-position-mobility, LER completers are not very mobile. The 
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ions^ 



two major types of employment positions taken by lERers art; academic positi 
universities and evaluation, development, and analysis positions in regional 
laboratories, national projects, school districts, or state departments of 
education, 'of the 10 former LER students who have left the positions they 
originally accepted, only three have changed career patterns, so to speak, by 
moving from .academia into evaluation and development, or vice-versa. A fourth 
represents a borderline case, as he moved from an adjunct faculty position to a 
full faculty position. • • 

Relatedly, it is of interest to note the pattern and trend of occupational 
emphasis that LER students have generated. In Table 2, the 25 LER Ph.D. 
recipients M the 2 completers without the degree have been grouped by period 
of leaving direct LER influence (that is, the "leaving date" column in Table 1) 
into Early (1968-1569), MiddTe (1970-1971), and Late students. Then, their 
present positions as given .1^ Table 1 are coded either professorial (I.e., 
faculty in a university setting) or evaluation, development, and analysis. The 
two persons with j^nt appointments (In evaluation with adjunct rank on a 
university faculty) are weighted equally in the two type-of -position categories. 
The greater emphasis on professorial roles is apparent in/the table, but equally 
in evidence seems a trend toward employment in non-professorial roles. The 
latter conclusion is obviously tentative, given the small n Involved, yet such 
a trend would seem understandable given the restricted etjiployment opportunities 
at present on university faculties. ^ 

Pattern and trend of occupational emphasis can also be examined in terms of 
total years of service by former LER students. Table 3 has been derived by 
using the Period of Leaving LER and Type of Position breakdowns as in Table 2, 
and by calculating years of service in each type of setting. The strange 
decimals reported result from the varying months of leaving by students and the 



Table 2: Present Positions of LER Completers Categorized by Students 
Period of Leaving LER and by Type of Position (as of 
December 31, 1973); " 



Type of Position 



Period of Leaving LER 



Professorial 



Eval uation, 
Development & 
Analysis 



Early 



968 
969 



4 

3.5 



\ 



7 or 78% 



2 or 22% 



Middle 



970 
971 



V 



5 or 65% 



or 35% 



Late 



(1 



972 
973 



1.5/*^* 



5 or 31% . 3 5^5.5 or 69% 



Totals 



16 or 59% 



11 or 41% 



) 
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Table 3: 



Years of Service of LER Completers Categorized by 
Students' Period pf Leaving LER and by Type of 
Position (as of December 31 » 1973). 



iti^f 



Period of Leaving LER 



Type of Position 



Professorial 



Evaluation, 
Development, & 
Analysis 



Early 




16.00)33 

17.33)""" 



.33 years 
78!t 



;.33 ) 9 
1.33 J""" 



66 years 
or 22% 



Middle 



{ 



1970 
1971 



"•"\l9.47 years 5.53lg 
5 0ojor66% .4.33J'"' 



_.86 years 
or 34% 



Late 



{ 



1972 
1973 



.33^ 



83 years 
or 29% 



2.92 K 
1.67 



59 years 
or 7U 



Total s 



54.63 years 
or 69% 



24.11 years 
or 31% 



16 



I 



% 

December 31 , 1973; cutoff date for calculations. As In Table 2, years of service 
in joint appointfi^nts are weighted equally In the two type-of-posltion categories. 
When total years of service in each type of position are thus considered, the 
. percentages reported in Table 3 do not differ dramatically from Table 2. 
Slightly over two-thirds of the 78 plus years of service by LERers have been in. 
professorial pursuits. The apparent trend toward Increased employment in non- 
professorial settings could, of course, rather quickly shift this balance. 

A final means of visualizing the pattern of present employment of -past LERers 
is presented in Figure 2. In that display, the current positions of 26 students 
are plotted geographically, by type of position (Dr. Larry Nelson's position in 
New Zealand is not shown). In the 1969-70 Progress Report and Evaluation of LER, 
a similar map (page 19 of the report) plotted the locations of the 15 students 
who had then completed the program. The information in Figure 2 can be compared • 
with that in the 1969-70 repor^^^^Tli^al lent points are: . 
(1) The heartland of the Midwest, Illinois and Indiana, have been vacated by 

LER in 1970, there were four LER representatives there, now there are 

none. ' . 

t2) LER has experienced its first placement" in the South. 

(3) LER is twice as international as in 1970 -- a Canadian branch remains as 
before, while New Zealand has been added. 

(4) LER has Increased its representatives on the Western seaboard from three in 
1970 to eight presently. 

{5) The upper-Mjdwest and plains states are more extensively covered than 
previously. 

(6) Seven recent placements have been made within 300 miles of the University 
(and six of these seven within 100 miles) only one such position was 

« 

held in 1970. 
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^* funding Support Level s 

In this final section of .the historical overview, data are presented on 
federal grant fur levels over the seven-year peri^d from September, 1966, to 
August, 1973. These figures, while meaningful in their own right, do not convey 
all of the resource support that was channeled into LER. For example, the 
figures given do not indicate University support for the LER program (such as 
professional salaries, in-kind contributions, university fellowships and teaching 
assistantships ana associateships for trainees, etc.), support for trainees from 
other sources (e.g., NDEA, some internship experiences, etc.). 

The federal grants made directly to LER are listed in Table 4. They are 
classified in two ways: grants for training (1966-1^73) and grants for program 
development (1966-1969). Further, the total grant award for each year and the 
accumulated grant awards over the seven year period are indicated. 

It might be noted that the grant award for 1972-73 is of a different 
character. The total grant award was for $99,188; $8,000 of this is not reported 
in Table 4 because it was to support a visiting scholar not directly connected 
with LER. The remaining $91,188 was utilized primarily for trainee support 
(stipends, dependency allowances, tuition, etc.) in both research and evaluation, 
instead of concentration on just research as formerly was the case. Secondarily, 
the award was used to develop certain products centering on LER training program 
procedures. Exemplar products developed included an evaluation simulation 
exercise, programed materials on experimental design, this history and evaluation 
of LER, evaluation instruments for research training programs, etc. The grant 
award for 1972-73 actually ran from June 1, 1972, to August 31, 1973, so that 
product development could commence during the Summer of 1972. 
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As can De noted in Table 4, the combined grant awards over the seven years 
accumulated to\ust over $400,000. The information contained in Table 4 is 
graphically presented in Figure 3. The relatively small amount granted for 
program de^lp^ent is easily ascertained in the figure, as is the dip in the 
grant,„^wffi^ for 1971-72 and the somewhat unexpected increase for 1972-73. A 
full explanation for tne 1972-73 increase would probably be better determined 
- by scrutiny at the USOE end of the transaction. It can be noted, however, that 
when. tl)e relatively smaller 1971-72 grant and the relatively larger 1972-73 
grant are averaged, the mean is quite cldse to the annual grants in preceedtng 
years. Finally, both terminations of funds can be seen in Figure 3 — termina- 
tion of program development funds as of August 31,-1969, and termination of 
training funds as of August 31, 1973. 
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II.' m UR TRAINING MODEL 

This section of the report contains a consideration of the major elements 
. that make up the LER training model. Opinions and judgments of past and present 
LERers about, the various elements are also integrated with'the description. In 
a sense, then, some sunmative evaluation concerns are addressed in this sec^tipn, 
**^Ntf|ile the final section of this report presents a sunmary of r sunmative evaluation 
considerations. 

No attempt is made here to describe every element in the training model, 
but rather only the major elements. Similarly, extensive description of 
elements selected is not undertaken, particularly when they have often been 
w scribed in earlier LER reports. HQwever, essential features of major elements 
are included in enough detail to permit the reader a reasonable conception of 
the topic under discussion. 

An expl anat6|^ote on data^ sources is in order. In addition to LER 
archives, data from the instruments found in Appendices A and B are utilized in 
Sections II and III. References to specific items are provided in the text. 
Further, data resulting from the questionnaire in Appendix A was tabulated in 
four time periods for LER students completing the program: 

(1) Early: Students completing in. 1968 or 1969 (n»8 returns out of 9 possible); 

(2) Middle: Students completing in 1970 or 1971 (n»10 returns out of 15 

possible); 

(3) Late: Students completing in 1972 or 1973 (n=7 returns out of 8 possible); 

(4) Present: Students presently enrolled, but commencing prior to September, 
1973 (n=7 returns out of 7 possible). 

"Completing" was defined as finishing the course work and leaving LER, whether 
or not the dissertation was finished. This definition was adhered to to increase 
the probability that a time-group (like "early" or "late") was reacting to the 



23 

same LER experiences, at least in regard to' time, when responding to the 
questionnaire. In most casefe, this categorization on the basis of time did not 
lead to dramatically different resul^ts, and, therefore, an overall average is 
often all that 'is reported. On occasion, the time-categories will be used to 
report the data. Finally, note that the two students commencing in LER in 
September, 1973, Cathy Bower, and Karl White, did not respond to the questionnaire. 

A. LER Staff ' 

A key element in the LER program is the professional staff. From, a solitary 
member in Sept'imber, IQS^ (Dr. Ken Hopkins), the professional staff gradually 
expanded to four by September, 1970 (with the addition of Drs. Glass, Goodwin, 
and Worthen)." As of Fall, 1973, LER staff is down to three with the departure 
of Dr. Worthen; possibly Dr. Paul Bradley will decide to^affiliate directly with 
LER over the comfng years. 

f 

In a very real sense, "professional staff" of LER can be most reliably 
interpreted in terms of Drs. Hopkins and Glass. Their continuing atteTrtion to 

« 

the development and maintenance of the LER training model does not need docu- 
mentation. As already noted (Table 1). together they have chaired the theses 
committees for 80 percent of LER graduates. Extensive interactions of Several 
types occur between these faculty and the students. The students as a group 
clearly identified informal faculty-fellow interactions, during collaboration on 
faculty-initiated projects, as the single activity of most educational value to 
them (Item 1, Appendix A). Further, the productive, active nature of the 
careers of Drs-. Glass and Hopkins, easily ascer^ined by examining their vitae, 
probably serves as a model for many LER students. 

In terms of other support personnel within LER itself, the positive 
advantages of having a strong, tolerant, flexible, and non-flappable secretary 
is well-established. LER has been particularly fortunate in this regard: Linda 
Venter, Anita Dunlevy, and Nancy Gallagher have each served several years with 
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distinction. Since September. 1973, Viki (VictoHa, for short) Bergquist, 
secretary for the Division of Foundations and Research Methodology, has also 
taken on a substantive support role for LER (schedulej coordinating, paper typing, 
client consoling, etc.). 

In general, after reviewing LER archives and questionnaire responses of past 
and present LERers, this analyst feels reasonably comfortable in identifying the 
LER pr^|fessional staff as the critical, salient element of the training model. 
The only other element approaching the importance of the professional staff 
appears t(| be the quality of student input. 

^' University Support and Location Within the University Setting 
; The supporting professional staff resource in the Universl'ty of Colorado 
comnunity is quite good. In psychology, mathematics, sociology , and other 
disciplines. LER students can find .substantive courses and experiences. Within 
the School of Education, there have likewise been some professional staff 
inclined and abie to work on an elaborated basis with LER students^ Th6 frequency 
of Jhis latter event occurring, however, has not been as extensive as would be 
desirable. It is as if the historical legacy of limited research activities in 
the School o^ Education has not been overcome. Although some notable faculty . 
member exceptions can be discerned in this regard and some overall Improvement in 
the School of Education detected, the fact remains that only limited opportun- 
ities exist for LER students to become engaged actively in orifloing research 
activity with School of Education staff. Most LER students Involved in research 
during their training become so In conjunction with the LER professional staff. 

With regard to resources^thei^ than personnel, the University has supported 
LER quite well. With ^e foresight [and wisdom to locate in Boulder, the 
Univc.-sity now p "^sents an attractivesjiosture to graduate students, and others, 
seeking relatively pure ecological and aesthetic settlags. Maintenance of this 



\ 

\ 

on 

f 

"advantage" remains to be seen. given tho growing number of days that even the 
Eastern slope has had a serious smog problem. The University, as well as other 
agencies in the metropolitan Denver area, has also given home to a number of 
curriculum development and research projects that occasionally become internship 
sites for LER students. 

More directly in terms of LER, the University has provided considerable 
program development support in the form of. professional salaries, some teaching 
assistantship money > and' physical space. The first home of LER was adequate 
given the initial small size of the program and subsequently was improved with 
the move to the present location in the Education Annex^ LERers have found such 
facilities to be quite adeq^iatc (overall average of 2^2 on a / to 5, adequate to 
inadequate scale; item 17, Appendix A). 

It is of note that LER's two physical locations have been facilities some-, 
what removed and remote from the School of Education proper. Without giviag 
undue attention to this factor of isolation, speculation as to the probable 
disadvantages and advantages'of such an arrangement is fun. The major disadvan- 
tage, at least in the perception of several School of Education faculty, is that 
the physical separation limits the accessibil ity .to and awareness of, LER 
services — this attitude was documented in the 1969-1970 LER evaluation report. 
This perception is not shared by LER professional staff, In fact, 'LER staff 
brieve that the major result of - ing housed in the School of Education itself 
would be an increase in casual conversation and visiting with other education, 
faculty, but a reduction in LER time resources available for productive 
conrulting with clients (Question B-11, Appendi^^. LER staff*, and this 
observer, consider it unlikely that locating LER in the actual School of 
Education building would lead to any real increase in productivity or research 
by education faculty and students. * 
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LER §tudents were also queried on this issue (item 20, Appendix. A). 
Although some variability in attitude is evident on this issue, LER students, on 
the average, neither strongly agreed nor strongly disagreed that closer ties be 
established between LER and the School of Education. This writer believes that 
the esprit developed by LER students, and in a sense the LER program itself, 
might not have occurred in the same intensity or form had the Laboratory been 
housed, from its inception within the School of Education. 

C. RejSfuitinerff , Requirements , and Student Quality 

,As indicated earlier, quality students exist as key elements in the LER 
training model. Strong students seem to have had the effect of requiring or, in 
a sense, of helping form, a strong program. Further, strong students have 
positively influenced and enriched the personal interactions taking place within 
LER. \ 

LER in its early years attempted formal recruitment procedures. This effort 
was highlighted by the somewhat standard, colored brochure describing the program, 
admission criteria, application deadlines, etc., that was mailed to all the 
standard places, primarily institutions of higher education. In the opinion of 
LER staff, the yield from following these standard operating procedures has been 
small. They believe now, and have for several years, that tweir most effective 
recruitment "device" is a group of knowledgeable agents around the country who 
know LER well and who recommend the program to promising students seeking 
graduate stuHy (question B-3, Appendix B). Over three-fourths of past and 
present LER trainees applied because an educational researcher or university 
professor had recommended LER, often after stimulating the student's interest in 
educational research and evaluation. Since the first year of LER's existence, 
there have been more excellently qualified applicants than traineeships. . y 
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The five major criteria for selection of UR students are indicated below: 

(1) aa undergraduate grade point average (GPA) of 3.0 or above and 3.5 in any 
' graduate work completed; 

(2) a combined Graduate Record .Exam (GRE) score of 1250 or above, with a 
Quantitative sub-score of not less than 625; 

(3) a background in mathematics; 

(4) a commitment to a career in educational research; arid . 

(5) strong letters of recomnendation (particularly from professionals knowing 
well the LER program). • 

Although not all" five criteria were met fully by all students admitted to LER 
(this was particularly true for the math background criterion), all students in 
competition for financial support were considered in terms of each of the five 
criteria. 

LER principal staff were asked to indicate the extent to which the various 
criteria were important in determining which students would be admitted to the 
Lab (question A-4, Appendix B). They agreed that GRE scores were weighted 
heaviest, followed by undergraduate GPA. T"iey were uncertain of relative weight 
for the other three criteria, ,but did indicate that a strong endorssment from a 
person knowledgeable about LER could mitigate considerably even the poor 
connotations associated with a GPA well below the established "minimum." 

Principal LER staff were'^also questioned as to th^ possible undesirable 
effects of the high entrance criteria, such as the charge of "elitism" (questions 
A-5 and B-4, Appendix B). It can be noted that these entrance criteria are 
markedly above criteria for other graduate programs in education at Colorado and 
most other institutions (and even above those for most educational research and 
evaluation training programs in the country). LER staff were unaware of any 
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substantial negative effects of the high criteria, although they thought such a 
connotation might threaten some potential usees. They saw no real substance to 
any elitism charge, and actually wished their program (as distinct from the 
admission criteria) might be vulnerable to such a charge, feeling it would be 
another basis for program esprit. One replied (somewhat facietiously, I believe). 
"If this is elitism, I don't ever want to see its opposite." There is no shortage 
oft one-1 iners around the Lab. 

The result of personalized recruitment and high entrance criteria can be 
ascertained by exam^^Fing)past and present LER students' performance on some of 
the criteria. (The data about to be presented do include Cathy Bower and Karl 
White; both commenced work in LER in September, 1973.) Relevant characteristic 
averagers, as well as average age, are presented in Table 5. 

The non-completers column in Table 5 refers to the five students who 
commenced the LER training program, but who terminated before completing, ftie 
left for personal reasons after only a summer in. the program, one left to assume 
an even lower profile with regard to the draft, and the other three were not 
performing adequately and were counseled to seek more appropriate professional 
objectives. Of special note is the fact that four out of the five non -completers 
entered LER during the first 10 months of the program's existence -- the fifth 
entered in September, 19^9, and terminated in May, 1970. This reduction in non- 
completers is quite dramatic. When asked ir*the non-completers had common 
characteristics (question A-1 , Apjjendix B),.LER staff indicated that they seemed 
to have had less comnitment and interest in the career implied by the program. 
In Table 5, their lower Verbal GRE (and hence their lower combined GRE) and 
their younger age are conspicuous. 

Other important features of Table 5 are the relative stability from period 
to period in most of the variables, the consistent advantage of a full standard 
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deviation of the Quantitative GRE sub-test over the Verbal GRE sub-test, the 
nearness of the average undergraduate GPAs to the alleged criterion of 3.00, and 
the older entrance age of the early completers. Incidentally, the generally 
e-xceptional performance of students once admitted to LER is easily documented by 
referring to previous LER evaluation reports. 

A final point, related to recruitment, that bears mention is program size. 
The proposal written for LER activity in 1967-68 Implied that the fully-opera- 
tional program would have 24 full-time students at a given time. .This level of 
trainee enrollment was never reached — the highest number of LER students at any 
point in time was 13 and occurred during the 1968-69 .and 1969-70 academic years. 
With the physical and human resources available to LER, it is hard to imagine a 
program as large as 24 trainees. When asked about the likely consequences had a 
24-student level been reached (question B-2, Appendix B), LER staff were hard 
pressed to imagine happy consequences except maybe justification for additional 
staff and greater service to School of Education faculty and students. Possible 
negative consequences envisioned included lowering entrance standards, losing 
esprit with so large a student contingent, and watering-down faculty attention 
for each individual student. The decision to limit enrollment may not have 
been a conscious one in tTie sense that the main limiting factor apparently was 
the availability of student-support funds — yery few LER students have been 
self-supporting during their traineeships. For the most part, students" have 
seen the stipends provided as adequate (item 18, Appendix A), but recent 
/nflation and the shift to job-oriented intern positions in Fall, 1973, appear 
to have made available stipends less adequate than previously. (On a 1 to 5 
scale, adequate to inadequate, early LERers registered 1.5,' middle 2.0, late 1.4, 
and present students 3.3.) 
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0' Experiences for LER Students 

One can hardly describe the total lab experience for LER students in the 
space of a few pages. The attempt here will begin by reporting the ranking by 
all LER students of various activities in terms of their perceived educational 
value (items 1 and 2, Appendix A). These rankings, presented in Table 6 by time- 
period group, are accompanied by an indication summarized for all students of 
whether each activity was underemphasized (-), overemphasized (+), or emphasized 
correctly by the student (0). Rankings and emphasis perceptions of the two LER 
principal staff are also presented. Further, the overall student average rank 
and the average staff rank columns in Table 6 have had the resultant new rank 
(based on the average ranks reported) indicated just to thfe right of the slash 
line in those columns. ' 

Several points can be made from the data presented In the table — other 
"findings" and^nterpretations can be achieved by the reader. First consider 
differences between the several groups of students. Differences between the 
three groups of completers {early, middle, and late) are not particularly 
dramatic. It does appear, however, that percept1ons^-«if the educational value of 
consulting and self-initiated projects increases somewhat over the three time 
periods, while the value of brown sack seminars and fellow-fellow interactions 
Is perceived to decrease. The early completers stand out for their high ranking 
of course work (1.8). while present students rank course work particularly low 
(7.5). In fact, the present student group ranks several categories substantially 
different than groups preceeding them. For example, present students rank 
comprehensive exams, faculty-fellow interactions, independent study or reading, 
and self-initiated projects as higher in educational value' than do other groups. 
Their low ranking of the dissertation is difficult to Interpret as most of them 
have not yet commenced that activity. 
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Overall student average ranks place the top activity in terms of educational 
value as faculty-fellow Interactions. Faculty, on the other hand,. rank that 
activity quite low iq educational value (6.5). Large discrepancies exist between 
students and faculty on other categories. Students consider fellow-fellow inter- 
actions to have more educational value than do the LER staff, wljile LER faculty^ 
see greater educational value than students in comprehensive exams, the disser- 
tation, and self-initiated research projects. By comparing this table to an 
earlier, similar-format one, produced in the 1959-70 evaluation (Table 4, page 31 
in that rqpprt), t|t clear that the average student rankings then and now are 
^)4te similar; while LER staff rankings have changed markedly on some activities 
(e.g., faculty-fellow interactions from 1 to 6.5, dissertation firom 4.5 to 1.5, 
^fe( low-fellow interactions from 3 to 9, and self-initiated projects from 7.5 to 
3.5). 

Perceptions X)f emphasis placed on the activities by students in general seem 
as one might expect (and are close to perceptions reported in the,i969-70 
evaluation, especially for students). Note that students as a gnpup did not feel 
overemphasis on any activity, w'-ile one of the two LER principal staff indicated 
overemphasis on consulting by s.udents. Students further indicated that they 
correctly emphasized most. activities, considering only independent study or 
reading, self-initiated pr^^cts, and (to a lesser extent) faculty-fellow 
interactions to be ju^Q^reijiphasized. The LER faculty agreed closeljpwith student 
perceptions except for course work ~ students perceived correct emphas s while 
faculty indicated underemphasis. (In 1969-70, faculty indicated correct 
emphasis on course work.) 

Finally, in considering Table 6, let us examine briefly each of the 10 
activities in order of listing as other dataware available bearing on some of 
^hem. 
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0) Bro'-zn Sack Seminars . Brown sack seminars seem to enjoy general disfavor, 
with hardly anywhere to go but up in terms of perceived educational value. In 
response to an interview question fitem B-9, Appendix B) , LER staff indicated one 
major difficulty was thf paucity of real research problems that were available to 
examine and to build ^signs around during the seminar. They stated further that 
the major objecti^/^f the brown sack seminars had become personal and social 
rather than instructional and content-oriented. Although admittedly not critical 
for training, some support for them was expressed in terras of their being helpful 
for esprit and good settings -*n which to discuss value orientations. 

(2) Comprehensive Exams . The second activity, comprehensive exams, was rated as 
low in educational value by both students and staff. A recent University of 
Colorado change has resulted in the elimination of first comprehensive exams that 
covered material in basic foundation areas. The rating by present students 
indicating a somewhat increased educational value in comps my be a manifestation 
of reaction to what formerly was second comps — exams over one's area of 
emphasis or speciality. Before, given the wording of the question, one had to 
react to botn first aid second comps together. 

(3) Consulti ng on Faculty -Directed Projects . The third activity in Table 6, 
consulting on faculty-directed projects, is rated relatively low in educational 
val\je by both students and faculty. This sometimes involves a substantial portion 
of a fellow's time (e.g., state assessment or AERA Task Force), but often is 
quite short in duration (Title III, ESEA, evaluations). 

(4) Consul ting " Off-the-Street ." Activity 4 in Table 6 may take the prize as 
the Veal sleeper" or "dark horse" of the original LER proposal written late In 
1965. In that document, ass isting^ faculty and students of the School of Education 
and other departments within the university on questions of research design, 
problem development, computer utilization, measurement, and statistical analysis 
Is discussed as an objective, but so are a variety of other Internship experiences 

i 
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in a number of settings. Over time, "off-the-street" consulting has mushroomed 
and assumed increasing amounts of LER students' time, and other internship 
activities less. For example, in 1968-69, a qrand total of 1 ,492 (Why does that , 
number make me think of Columbus, Ohio?) hours of free consulting was provided 
to University of Colorado graduate students by LER. Slightly over half of t|iese 
hours were spent with School of Education graduate students. Further, 473 hdurs 
of assistance were provided to university faculty. \ 

Because of the increasing demand and press for this service, s'^veral related 
steps had to be taken. During the 1968-69 academic year, the internship expecta- 
tion for LER students was reduced from 600 to 400 hours annually. Further, 
either D,r. Glass or Dr. Hopkins began sitting in on the initial interview with 
new clients, to speed the process and to increase the instructional aspects of 
the session for the LER students. These steps were taken to help maintain a 
better balance in the students' academic diet, and the first step was, allegedly, 
to encourage more accurate logging of internship hours. It became necessary to 
restrict tiie avdi lability of the service to School of Education fa' Ity and 
students, excluding requesting students from speech, audiology^ communication, 
anthropology, etc. Further, attempts were made to reduce the tendency for LER 
students, normally in desperation, to run data analyses for clients rather than 
to explain procedures for the nth time in hopes that insight would strike the 
fumbling client. Related to this latter effort, LER staff reported (question 
B-7, Appendix B) less actual analysis of data by fellows now than during the 
early years of LER. However, off-the-street consulting continues to involve 
about one-third of the fellows' available time, data analysis still makes up 
over half of this consulting load, with the remaining consulting time split three 
to one, education students to education faculty (item 31, Appendix A). 
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Fairly consistently over the years, LER fellows have viewed the clients in 
these consulting sessions to be ill-prepared (3.7 on a \ to 5 wel 1 -prepared to 
ill -prepared scale), somewhat grateful for the service, and strongly benefitted 
•jy it (item 14. Appendix A). As implied in the proceeding paragraph, LER students 
found tJismselves assuming too much of the client's data analysis work (item 15, 
Appendix A) . 

This service component is large and. in the view of at least one principal 
LER staffer, represents a serious weakness in the training model. What proportion 
of this consulting represents solid training for the student, as distinct from 
pure service as when the LER student repeats for this client, as he did for six 
previous clients, the meaning of the assumptions underlying ANOVA? Still, the 
relatively high ranking extended it by both students and staff (Table 6) suggests 
it is a valuable training element. In question, apparently, is not its value as 
alegitimate training activity, but rather the massive time requirement that it 

0 

necessitates -- and the feeling that some of this time (particularly that which 
represents re-hoeing the same field) might be better spent on other activities. 
The' evaluation of such a heavy time commitment for service-oriented off-the- 
strjeet consulting might be turned siround given the present level of awareness of 
its time-usurping character and the concurrent necessary to find financial support 
for students. This latter requirement has already ca^jsed a return, for some 
students, to the earlier conception or internship as work-activity for a research- 
or evaluation-related agency and, necessarily, a reduction in those students' 
available time for off-the-street consulting. A very real concern, however, is 
avoiding the low-training-yield internships all too comnon in the early experi- 
ence of LER. Key factors in this regard appear to be having the intern working 
on a worthy problem and being directly monitored by a research-valuing and 
research-understanding supervisor (question A-8, Appendix B). 
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(5) course work. Course work, the fifth activity in Table 6, v-'ss ranked very 
high in educat^Mal value by both staff and students (with the ex..)v ;on of 
present LER trainees). Course w(Jrk is usually taken to mean the 11?. "^re 
Curriculum" in research methodology and sl^atistics. The Core Curricui.?... in 
turn; is usually operationally defined as the courses listed below: 
Education 



El em. Stat. Methods • 
Adv. Psych. Foundations of Educ. 
Inter. Stat. Methods 
Adv. Educ. Meas. & E!val. 
Adv. Soc. Foundations of Educ. 
Eval. of Sch. Systems & Programs. 
Methods of Educ. Research 
Exper. Design & Analysis I 
Exper. Design & Analysis II 
Internship in Educ. Research I 
Inte-nship in Educ. Research II 
Internship in Educ. Research III 
Internship in Educ. Research IV 
Problems in Eval . 
Master's Thesis 
Doctor's Thesis 



480 (3; 
503 (2: 
505 (3) 
611 (3) 
516 (2) 
591 (3) 
600 (2) 

604 (3) 

605 (3) 

608 (21 

609 {2T 

610 (2) 

611 (2) 
695 (2) 
700 
800 

Psychol ogy 

587 (4) 

588 (4) 
691 (3) 

Mathematics 

481 (3) Probability Theory 

Further, it can be noted that this is not ahard core, but a flexible one. 
courses are added and dropped based on evaluative data. Also, the courses taken 
by any one fellow depend pn his objectives and interests.. Finally, since about 
one-third of .the fellows do not aajor in research methodology but In other sub- 
stantive areas of education (e.g., educational psychology, guidance and counseling, 
•sfcience education, mathematics education, etc.). their emphasis on the flexible 
core curriculum is sometimes less pronounced. 



General Statistics I 
General Statistics II 
Multivariate Analysis 
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Past amd preseirtrtse students and current pi;incipal LER faculty were asked 
to evaluifte the core currlaulum and tha emphasis on evaluation methodoloqv in the 
cujxmilum (items 25 and 2b, Appendix A). Their responses appear in Table 7. 
One is Immediately struck byHhe great similarity across the time-period groups. 
In terms of the first four-itemi, the early completers group viewed the core 
curriculum in the most fav^i^ole manner (that is, perceiving a balance on the • 
amount of math included a/id amount of freedom allowed, reporting a near-balancfe 
on the theoretical -applied continuum, and rating the design of the curriculum 

f 

higher than all later groups). Faculty perceptions were close to student percep 
tions although faculty reported the curriculum less well designed than students 
did and also felt more strongly the lack of emphasis on evaluation. 

In interview question's (A-11 and B-13, Appendix B), LER staff reported a 

i 

gradual trend in the curriculum and experiences toward evaluation, but not at the 
expense of having to reduce emphasis on pther portions of the curricultiri. The 
staffers agreed that students were taking\fewer offerings in the behavioral 
sciences now than previously. They disagrefed, however, as to whether the cur- 
riculum allowed too little or too much freedom and whether a return to more firm, 
structured course requirements was wise or warranted. The disagreement extends 
to independent study, with one professor proclaiming 'unilaterally the need for 
more structured, guided, and documented independent study. This debate likely 
will continue. 

Another point might be presented before leaving the topic of course work 
some fellows express the need for e{pertis2 in a content area (in addition to 
research methodology). Of course, about one-third of LER grads do have content 
area emphases and sophistications, such as those majoring pr garnering an 
additional major in educational psychology,, guidance, etc. In the original pro- 
posal establishing LER, it is clear thali the Intent was "to emphasize hoth the 
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substantive and methodological aspects of research" (p. 10). This intent has not 
been fully met, in large part due to the limitad time available for LER fellows 
to pursue a substantive content area while still mastering the research methodo- 
logy area. This lack is apparently felt deeply by only some LER students and can 
be ascertained by a questionnaire response (item 33, Appendix A). Asked whether 
they felt inadequately trained in a specific discipline or content area, students 
responded on the average at 3.4 on a 1 to 5 scale froij} "to a great extent" to 
"not at all." Going to individual responses, only one LERer responded with a. "1", 
while nine others indicated "2," 

LER staff a^ree that lack of a content area sophistication other than research 
methodology can be a detriment to some students, and they do recognize such pre- 
paration as a LER program objective. However, they feel two conditions would be 
necessa.. to allow real Improvement 1„ this .r^r.. , they indicate a need 
for additional educational faculty, with on-going research interests in instruc- 
tion, educational psychology, etc., who could provide nurture and guidance for 
LER students in such content areas. Second, they believe that students would 
have tc^ be willing to spend additional time as a student to learn and integrate 
the new content emphasis and material. How likely these two conditions are to 
occur is difficult to say. LER staff do believe, however, tiht former students 
who suggest that lack of a content area specialty is keeping them from publishing 
research may often be efrectihg a strawman to rationalize their own limited 
research productivity. 

(6) Dissertation . The^issertation is viewed as having more educational value by 
LER staff than by past and present LER students. In general, past students see 
dissertation guidance by Drs. Glass and Hopkins as substantial and appropriate. 
For example, students rated their guidance 3.3 on a 1 to 5 scale from restrictive 
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to vague (item 5, Appendix A). Other advisors, incidentally* either only 

» 

partially affiliated or unaffiliated with LER, were rated 4.5 on the average on 
this same item. 

' It is appropriate here to discuss the interdisciplinary aspects of the LER 
program, as some relevant elements occur at the dissertation stage. (Likewise, 
this topic could have been addressed under the preceeding section on course work.) 
The intent of the original proposal was to provide a rich interdisciplinary 
exchange for students in terras of course work, informal seminars, and diiserta- 
tion efforts. This was achieved to a substantial degree in the area of disser- 
tations, particularly with -participation by psychology professors, but less 
recognizable with regard to course work. Ir^ a sense, the interdisciplinary 
aspects of the LER program seemed to be more an expectation desired and sought 
rather than a spontaneous, viable happening occurring in the program. In sum, 
however, interdisciplinism in LER reached levels unusual for doctoral programs, 
but still fell below the rich, fertile exchanges originally envisioned. 
(7) Facul ty«f el 1 ow Interaction . In Table 6, this activity was ranked highest, by 
past and present LER students, in terms of its educational value for them. Con- 
si stent with this high ranking were student responses on several questions (Items 
3, 4, and 6, Appendix A). Drs. Hopkins and Glass were rated as easy tq see on 
professional matters (4.7 on a 1 to 5 scale from "difficult to see" to/"easy to 
see"), as providing neither restrictive nor vague course, degree, and [disserta- 
tion guidance, and as providing "a great deal professionally" to students (1.25 
on a 1 to 5 scale from "strongly agree a great deal professionally was provided" 
to "strongly disagree"). 

^ Many types of activities can occur under this category. For example, 
explanation by faculty of School of Education and Graduate School requirements 
could, and on occasion does, occur. Students, by the way, often are directed by 

/ : 
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faculty to other students to get "requirement questions" answered; this practice 
evidently is not too offensive to students (on item 7, Appendix A, students 
neither strongly agreed nor disagreed that LER faculty should know more about 
such requirements). Another, example of facu\ty-fellow interactions would be work 
with non-LER faculty on faculty-initiated projects LER students, on the average, 
reported knowing well two to three education professors with whom such efforts 
might be undertaken (item 19, Appendix A). However, fo^ many reasons previously 
cited, this orientation of students when ranking the 10 activities in Table 6 
would have been unlikely given the low frequency that such events occurred. 

^tost likely when past and present LERers made their rankings, they had in 
mind direct interaction between themselves and Drs. Glass and/or Hopkins on 
professional and methodological concerns. Clearly there is not enough of these 
two men to spread too far in this regard. Five or six quite active LER graduate 
students are advised by each man — this in and of itself is a large time commit- 
ment. Further, each man has spent considerable time with non-LER students, both 
during initial client-intake interviews and in other sessions assisting them on 
their dissertation planning. Further, each man has consulting activities with 
faculty, with agencies nationally, and the usual university teaching, research, 
and other service obligations (e.g., committee-work) to meet. Still, from all 
the data this evaluator has been able to find, collect, and interpret, the 
extent of direct faculty-student interaction for LER students apparently exceeds 
such interaction opportunities in most other doctoral programs in the university. 

Interaction between LER faculty and LER students is not just a matter of 
putting aside a certain number of hours to meet together (item B-8, Appendix B). 
Rather, substantive Issues or tasks afe usually the focus of such encounters. 
Further, there are issues surrounding the nature of how such interactions should 
proceed. For example, should faculty take the lead role, thus possibly increasing 
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the "efficiency" of the interaction in tenns of information imported, but also 
increasing the likelihood of student dependence upon tV faculty member — or 

should faculty be tolerant of inefficiency in such a setting, opting for such a 

. ^ ■ \ 

posture to increase the training potential of such sesslon^x for students? Such 
"inefficiency," of cou/se, lengthens the time commitment for faculty. 

In sum, this activity blends and^^^^egrates and places 1n\^teraction the 
two key components in the LHR tra1rt1|i£^ifedel — that is, the LER ^^udents and 
the principal LER faculty. The high ranking given it by students arid the plunging 
ranking given it by faculty (as plunging compared with rankings four years 
earlier) suggest that careful attention needs to be given to this activity at 
once. Its maintenance and viability appear critical for the success of the LER 
training model. Interestingly, one LER professional staff member indicated his 
hope that such interactions could lead to even more faculty^student collaborative 
research. Such a practice was more obvious in the earlier years of- LER than 
recently, and obviously would require even additional faculty and student time 
commitments above the present high level. * 
(8) Fellow-fellow Interaction . Activity 8 in Table 6 is also important, partic- 
ularly in the perception of st,udents. Three questions (items 10, 11, and 12, 
Appendix A) dealt with fellow-to-fellow. Interactions. Past and present students, 
indicated much orientation of new fellows by older fellows (1.8 on a 1 to 5 scale 
from "a great deal" to "very little"). Help was extended between fellows partic- 
ularly in regard to client problems and course planning, and (to a substantial 
but lesser extent) on actual course work, degree requirements, and professional 
problems. In all, fellows perceived informal iriteraction between themselves as 
quite valuable (1.4 on a 1 to 5 scale from "valuable" to "not valuable"). 
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{9) Independent Study or Reading . This activity was ranked of moderate educa- 
tional value by students and staff. Note also the high incidence of students 
and -itaff perceiving independent study and reading as underemphasized by students. 

One wonders why present students ranked it so much higher than previous 
students or LER faculty. Possibly, the age of independent study ("everybody do 
your own thing") is upon us. More likely, present students still have hopes of 
pursuing independently many topics, while past students and faculty are more 
aware of how rarely time is found for such independent activities. Intents seem 
to far exceed actual transactions in regard to independent study and reading. 
(10) Self-initiated Projects . Self-initiated projects, like independent study, 
were perceived by both faculty and students as underemphasized by students. - 
Faculty ranked such projects considerably higher in educational value than did 
students. One interview question (item A-9, Appendix B) provided related data. . 
Both LER principal staff were receptive to the idea of requiring LER students to 
conduct a small research project from conceptualization to completion. Recalling 
the formerly- funded research mini -grants, tliey indicated that the experience of 
following a similar model, possibly on a reduced scale, might prove quite educa- 
tional for students. (Relatedly, LER staff did not see as particularly necessary 
training LERers to develop and write proposals £er S£ this was considered a 
fairly routine skill fairly easily learned if the demand arises; question B-15, 
Appendix B.) 

It is appropriate at this juncture to introduce another potential experience 
for students. Although not listed explicitly in Table 6, the experience of 
publishing is implied by activity 10. No attempt will be made here to treat the 
many arguments and facets thereof that have revolved for decades around the 
"publish or perish" issue (or, as it was known long before the invention of the 
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printing press, the "chip or ship" issue I couldn't resist putting that in, as 
I am comfortably aware that I will probably be the only reader ever of this page). 
Rather, the potential training value of such an experience will serve as a focus 
ibr discussion. 

Anyone hoping for a clear resolution of thi^s training value of publishing 
might as well skip this paragraph. In fact, neither the LER students nor the LER 
staff are of one mind on this issue. When asked to what extent it was Important 
for LER students to publish formal papers during their training program (item 39, 
Appendix A), averages on a 1 to 5 scale fr'bm "to a great extent" to "not at all" 
were 1.9 for early completers, 1.6 for middle, 2.1 for late, and 3.0 for present 
students. On the same item, principal LER faculty responded 1 and 3. Thus, the 
training value is agreed upon, but not whether ^he value is great or only 
moderate. 

The evaluator had some visions of partially answering this question via a 
careful vita by vita analysis, looking at publications during the LER student 
years and then at publications since. The question to be addressed was whether 
those students publishing extensively while Lab students continued to be produc- 
tive later, as compared with career publication records of persons who did not 
publish extensively as Lab students. So much for good intentions ofCli. 
evaluator. My subsequent analysis would suggest that high publication produc- 
tlvity as a LER student tended to be associated with high publication produc- 
tivity in later career work, but the analysis suffers from several problems. 
First, nearly 30 percent of past LER students sent no vita at all (which in and 
of itself is possibly a general indication of lack of inclination to publish). 
Second, those vita that were received were not "standard" — some were up-to- 
date, others were not, some included a meticulous account of publications of all 
types, others did rwt, etc. Third, the back-up questionnai'-e item was often 
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'selected as the "item most deserving of not getting answered" by several of those 
respondents who hacTnot provided a vita, while other respondents merely 
referenced their vita which sometimes was current and complete but other times 
was not. To a large extent, then, the question remains unanswered. 

ftoving back to the previous point of the merit of LER students conducting 
complete mini-research projects while in training, it can be noted that such an 
experience would presumably include publication and therefore might encourage 
later productivity. At a more basic level, however, it would appear that this 
general issue might be addressed by present LER faculty and students. Of <ill 
student groups, the present "crop" ascribes least importance to publishing while 
still students -- with two faculty, one apparently in agreement With the students 
and one not, the debate should be lively. It might be possible, after extended 
discussions, to list under what conditions publication while a student should be 
encouraged (e.g., for those planning a university-based career, etc.). 

One final experience for LER students deserves special note — teachings 
The opportunity to teach in a university setting is considered by LER staff as 
very important, especially for students planning on a college or university 
teaching career (item A-10, Appendix B). It exists as an important element for 
such students, both in terms of their learning and their subsequent employability. 

At the University of Colorado, it has become increasingly difficult of late 
for students to receive full responsibility for teaching a class, particularly 
if the class is offerred for masters or doctoral students. Opportunities do 
exist to tutor and to assist professors in conducting a class. Pr^ently, some 
students arc afforded the opportunity to teach the elementary statistics course 
(480) and the basic educational measurement course (412). It would seem important 
to keep this experience available for all students desiring it. 
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E. Duration of Training 

The length of time that individuals are directly affiliated with LER in a 
student status can be derived from Table 1 presented. earlier. The period is 
usually two or three years, usually depending on wither the entering student has 
or has not already obtained a masters degree. 

Some past students in their questionnaires indicated a feeling of rushing 
through their program to meet staff expectations. Further, they believed an 
additional year in LER might have been beneficial to them, by allowing more work 
in a subject content area, permitting initiation of publications, etc. Whether 
these good things would have occurred or not is uncertain, for, in response to 
concerns of early completers, LER staff began encouraging entering students to 
spend three years in L€R, whether or not the ^tasters degree had been earned 
previoasly. Even now, however, the concerns of "rushed" fellows indicated above 
seem often to be expressed by "unrushed" fellov/s. Possibly this is another 

« 

paradox of nature no matter how long the doctoral training program is designed^ 
to take, it will seem one year too short to students. 

A related topic is the attitude toward, and frequency of, leaving LER 
without the dissertation being completed. The attitude of LER staff Is clear 
enough they are against the practice. The frequency of leaving before 
finishing the dissertation is expressed in Table 8; several attempts at plotting 
the data did not result in improved insights about the practice and thus are not 
Included. Although one is tempted by the 1973 data to conclude that the early- 
leaving phenomenon is on the increase, this is probably unwise given the small 
n involved. Further, 1971 app'lpars not so very different from 1973. Additionally, 
the time period between leaving and completion is also of interest, and two of 
1973's early-leavers completed the Ph.D. before the calendar year ended. 
However, the data may be' indicative of a trend, particularly with the reduced 
availability of student support funds, the dramatically Increased cost of living. 
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Table 8: Number and Percent of Students Leaving LER 
Before and After Ph.D. Completed. 



Leaving LER Leaving LER 

After r.efore 
Year Ph.D. Completed Ph.D. Completed * 



1968 


3 (75^5) 


1 (25%) 


1969 


4 (80%) 


1 (20%) 


1970 


4 (67%) 


2 (33%) 


1971 


2 (50%) 


2 (50%) 


1972 


2 (67%) 


1 (33%) 


1973 


2 (40%) 


3 (60%) 



Totals 17 (63%) 10 (37^) 



All persons^! isted in this column have compVeted the 
Ph.D. (as of 12/73) except one student leaving LER 
in 1972 and one student leaving L£R in 1973; see 
Table .1 . 
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and the availability of close, professional enrollment (three out of the past 
four early-leavers have been employed at National Assessment in Denver). This 
index bears watching. 

Placement after Training 

Many aspects of placement have already been addressed, such as the type of 
position taken (Table 2), the years of service by type of position thus far 
provided (Table 3), and tne 'geographical location of positions by type (Figure 
2)- 

« 

Past and present LER students were asked to indicate by ranking whether their 
occupational preference tended toward developinent , evaluation, or research (item 
29, Appendix A). The results, presented in Table 9, show a clear preference 
for research, then evaluation anui^evelopment . Support for evaluation careers 
may, however, be on the increase. • 

Former and present students also ranked possible occupations and gave their 
percepti9ns of how LER staff would rank such occupation settings in terms of 
appropriateness for LER graduates (items 27 and 28, Appendix A). Their estimates 
of LER staff ranks were amazingly accurate, so much so that the actual data are 
not reported for fear that collusion might be charged. The preferences of LERers 
themselves are reported in Table 10 by time-period group. Of note are the rather 
high ranking given research anv. teaching in a small college or university, and 
the low ranking given a coordinator's job in a state department of education. 
Even more irroortant, particularly considering the trend toward less employment 
of LER grads in professorial roles (Table 2), is the clear preference given to 
professorial positions. In Table 10, research and teaching in a large and small 
university are ranked 1 and 2 by all groups, while. the early completers, present 
students, and all students on the average rank "teaching only in a university" 
as 3 (and the other two time-period groups rank it a near-third). These data 
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Taple 9: Average Rank of Occupational Areas by Time-Period 

Groups of LER Students 



Average Rank by Time-Period Group 



Occupational 
Area 



Early 
Completers 
(1968-1969) 



Middle 
Compl eters 
(1970-1971) 



Late 
Completers 
(1972-1973) 



Present Overal 1 
Students Student 
(1974-?) Average 



Research 

Evaluation 

Developrhent 



1.8 
1.0 
2.4 



1.1 

2.2 
2.7 



1.1 



2.3 



2.6 



V ;;3 
2.6 
2.3 



1.5 
2.3 
2.5 
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would suggest that the recerit trend toward employment in 'non -professorial settings 
is due more to the job market than to any dramatic change in graduating student 
Interest. Interest in employment quite otten takes precedence over interest in 



interests. / 

Finally, with regard to placement, it might be noted that LER professional 
staff believe that completing LER students have been placed well (question B-5, 
Appendix B). The name of LER. and the reputations of Drs. Glass and Hopkins, have 
not 'hindered or been detrimental to the placement of any fellow -- in fact, they 

^ 

have probably been instrumental in securing many placements for fellows. 

G. Changes in the LER Training Model over Time 

A final topic rounds out this description of the LER training model. Several 
changes have occurred in the LER Xraining model in the course of its seven-year 
history — many of them have already been alluded to. . The primary ones are 
listed below (not noted are the obvious early changes due primarily to the 
\program development grant, for example, the creation of new courses): 

— ^Shifting- emphasis from Internships of the apprenticeship-variety with applied 
"agencies (e.g., public schools, state department of education) to consulta- 
tion with individuals and groups doing research projects ( "off-the-street" 
consulting) . 

-- Reducing the annual internship requirement from 500 to 400 hours per studerj/t. 

— Reducing the number of required "core" courses, placing them in an elective 
category. 

— Having a faculty member sit in on client intake interviews. 

~ Increasing the involvement of LER students on faculty-led projects (e.g.. 
State Assessment, Title I, Task Force, etc.) and evaluations. 

A^number of lesser changes of an administrative nature also occurred (e.g., 

making a fellow the coordinator of client appointments, making another the 

coordinator of brown sack seminars, etc.). 



o 
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Four important features of the primary changes can be noted. First, they 
are not earth-shaking or particularly revolutionary. They seem to be well- 
considered responses to fairly wel 1 -documented problem areas. Second, they have 
been of long-standing. That is, the change once implemented has persisted. 
(However, the change involving a shift of internships from applied agencies to 
off-the-street consuUif^g may change again, as previously noted.) Third, the 
changes are direttly related to the student experience. Fourth, and significantly, 
the changes have primai^ily been the result of evaluation data, particularly from 
students. The change that is the exception to this general pattern is the 
increasing involvement |of students on faculty-directed projects. 

The earlier statement that these changes have been neither earth-shaking nor 
revolutionary does not imply that other more dramatic changes have not been 
considered. For example, three outside consultants (Art Lumsdaine, Sam tfessick, 
and Sam Sieber) evaluated LER iri 1971. One proposal they put up for consideration 
was the possibility of imbedding the training of LER students within a large- 
scaled on-going research project — learn to be researchers by doing research was 
the basic suggestion* The LER professional staff rejected this' notion then, and 
still do. Although noting the potential student financial support advantages of 
a large-scale continuing research project, LER staffers believed then (and now) 
that variety of experiences for trainees would be substantially reduced and 
students would be subjected to mainly low-yield experiences from a training 
standpoint (question A-8, Appendix B) . Past and present LER students are of the 
opinion that they would be less well -trained if LER had been focused around a 
continuing large-scale research project, 4.1 on a 1 to 5 scale fr*om "better" to 
"poorer trained overall" (item 32, Appendix A). 

This positive analysis that describes LER charge policy as relaxed, con- 
sidered, and data-based U not meant to imply that analysis should cease or that 
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significant changes should not now be considered. In fact, this evaluator feels 
that the opposite is true. Given the changing job market situation, the reduced 
availability of financial support for students, the changing perceptions (and 
possibly interests) of LER principal staff, and other factors, this observer 
believes that this is an opportune time to re-examine the LER training model, 
component by component. It is also an appropriate time to re-examine tihe • 
original objectives of LER to ascertain their present viability, merit, and 
relevance. ^ 
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III. SUMMARY SUNMATIVE EVALUATION OF THE LER PROGRAM 

As stated in the introductory section of this report, this writer is not 
unbiased in so far as LER is concerned -- rather, I see it as an effective, 
substantial program of high merit. Therefore, the reader is instructed that this 
final suimary section is best considered an internal summative evaluation. 

The "summary" label of this final section is deemed appropriate in that a 
number of summative type observations and judgments have already been made in 
both Sections I and II, particularly in the latter. In this section, probable 
effects ^on major consumers of, and organizations related to, LER will be examined. 
"Probable" serves as a suitable caution given the one-group ex post facto nature 
of this evaluation. A final, closing discussion then^ briefly considers the 
strengths and weaknesses of LER and makes a few reconmendations for future 
operations. . 

A. Probabl e Effects on the University of Colorado 

The major probable effect of LER on the University of Colorado is quite real 
and tangible* namely the program development effort. As already discussed, 
several new courses were added, existing courses were revised and altered 
drastically, computer applications became apparent, etc. At best, any prior 
program area that might have been termed "research methodology" was marginal. 
Now, however, LER is recognized locally (as well as nationally) as a strong, 
vital program in the School of Education -- many observers (including this one) 
consider it tiie strongest educational program in the School of Education. This 
development and rise to prominence occurred in a relatively brief period. 
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In a less dramatic way, one can also consider the federal funds that were 
received by the university for program development and trainee support — the 
accumulated total of over $400,000 in seven years is substantial. ^ 
_^ Finally, another probable (and very real) effect of the program has been 
establishing and maintaining a particularly effective team of research methodolo- 
gist nurturers. The team of Glass and Hopkins has had substantial and positive 
effects on an ever-increasing number of students, faculty, and other members of 
the educational community. (Although not as well known as Rowan and Martin, their 
humor is cleaner and substantially less funny.) 

B r \probab1 e Ejects on University of Colorado Facul ty and Students 

Responses to interview questions (A-6, A-7, B-7, and B-10, Appendix B), 
questionnaire items (30 and 31, Appendix A), and data in the archives all 
suggest that LER has had a substantial effect on some personnel within the 
University. 

Particularly and positively influenced have been School of Education graduate 
students. Via course work, research internships, and personalized consulting, the 
quality of theses for such students and their general research sophistication has 
undoubtedly advanced over pre-LER levels. This effect is likely substantial and 
should not be minimized. 

The influence and impact of LER on School of Education faculty has been 
disappointing and markedly below hoped-for and intended levels. Although several 
have monitored basic research courses, research productivity has not changed*' 
substantially. Although consulting with education faculty has increased somewhat 
over the LER years, such assistance seems to facilitate the professor's work, 
but the work typically is not a research product. There are, however, a few 
School of Education faculty who have been directly and materially assisted by 
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LER in doing substantial research -- these xceptions stand out. Further, many 
School of Education faculty now advise their graduate students to use the 

0 

facilities of the Lab. This advice to students demonstrates awareness of the 
service, but at times is irksome as research methodology instruction for the 
professor himself via consulting or course-monitorinq would seem as, or more, 
appropriate. 

LER' s probable effect on faculty and students outside the School of Education 
has been less Substantial than for \9ducational faculty and students. As already 
noted, the press for consulting services forced LER to limit access to the 
service. Consulting off-the-street with such groups correspondingly has dwindled 
considerably. Some students from other departments (e.g., psychology) still are 
indirectly effected by enrollment in courses taught by LER staff. 

C. Probable Effects on LER Students 

The primary most visible product of the LER program are the LER students 
themselves. First and foremost, the program was initiated to develop persons 
"to stimulate and conduct research studies focused on important aspects of 
education." 

Perceptions of LER staff (questions B-5 and B-17, Appendix B) and past and 
present students (items 16 and 34 "through 38, Appendix A) for the most part 
indicate positive and substantial effects of the program on LER students. 
Students who have completed agreed that the program met their expectations. On 
balance, they perceive themselves as better trained than their present non-LER 
colleagues (1.7 on a 1 to 5 scale from "superiorly trained overall" to "infer- 
iorly trained-overall") and ascribe this better training posture primarily to 
LER (4.4 0"^ a 1 to 5 scale from "not at all" to "to a great extent"). Further, 
asked to speculate how successful they would now feel had they completed a 
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graduate program other than LER, they indicated probably less successful (1.8 
on a ] to 5 scale from "less successful than I now feerl" to "more successful 
than I now feel"). They did not exprepsa great need for retraining at this 
time (2.3 on a 1 to 5 scale from "not k alll " to "to a great extent"), but did 
Indicate a need for further training (2.5H(n^a 1 to 5 scale from "to a great 
extent" to "not at all"). Areas mentioned most frequently for necessary addi- 
tional training were, in order of frequency, multivariate analysis, advanced 
statistical techniques, and evaluation. 

So, even with the limitations of testimonial self -report data, let us assume 
that the LER training program did have substantial effects on Its graduates. A 
second -question then looms significant. Did the obtained effect address directly 
the principal objective of the training program, that is, developing persons to 
stimulate and conduct research' studies focused on important aspects of education? 
The an^v?fer to this question Is not easily obtained. Can a case be made that many 
LER graduates are more service-oriented ("service" in the sense of assisting 
others to conduct their studies better) than research-oriented? If so, does such 
service constitute an important element of the "stimulation" sought in the major 
objective? Is the present LER training model too service-oriented -- how often 
do trainees see and work with Drs. Glass and Hopkins when the staff are Wearing 
their service hats? -- their research hats? — their evaluation hats? Is the 
desired LER graduate a research-methodology service-technician or a scholar? 
These are difficult questions meant for staff and student reflection, particularly 
appropriate if questions of possible new directions are examined. Some informa- 
tion in the section immediately following also bears on the questions raised. 
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^* Probable Effects on th£ Field of Educational Research 

A series of interview questions (A-16, A-18, B-17. and B-18, Appendix B) 
and items on the questionnaire (40 through 41, Appendix A) addressed the question 
of likely effects of LER and its graduates on the field of educational research. 
Former students did not particularly believe that they were producing research 
critical for the field (2.5* on a 1 to 5 scale from "not at all" to "to a great 
extent"), but they did see themselves working in major problematic areas in 
education (e.g.. integration, accountability, etc.) to some extent (2.5 on a 1 to 
5 scale from "to a great extent" to "not at all"). 

LER faculty perceived students more involved in major problematic areas than 
did the students themselves. One staffer expressed his opinion that although 
several LER graduates had become involved in a problematic area, it was likely 
an accident of employment and not due to any feature of the training prbqram. 
Incidentally, LER staff in citing examples of former students with heavy 
problematic-area-involvemept mentioned only fellows employed in non-professorial 
rol es . 

On the question of whether g^duates were producing research critical for 
the field. LER staff split (2 and 4) ."-Critical" sets a high standard and, in 
balance, one can point to few products of LER graduates that meet such a criterion. 
LER staff felt that possibly two or three of the quarter-hundred LER graduates 
might eventually produce such critical research. The areas mentioned as most- 
likely-to-be -produced were primarily concerned with research methodology rather 
than a content-area discipline. This observer agrees that little has been 
produced by graduates to date that merits the "critical" label. 

V 
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E. Overall Probable Effects of the LER Training Program 

Data of relevance here came from the archives (e.g., AERA Task Force Report 
No. 16, never published, but years in the PERTing that compared research training 
programs), interviews (question A-17, Appendix B). and questionnaire feedback 
(items 42 and 43, Appendix A). Compared to other programs of a similar nature , 
LER was perceived as clearly above average (4.5 by staf/f and 4.1 by students on 
a 1 to 5 scale from "distinctly inferior" to "distinctly superior") The list 
of programs cited as superior to LER (e.g., Stanford, Berkeley, Chicago) was 

stwrt. This observer would place LER in the top decile of research training 

/ 

prt)grams in the country. 

Compared to doctoral programs generally, the LER program was regarded even 
more positively (4.5 by both staff and students on a 1 to 5 scale from "distinctly 
inferior" to "distinctly superior"). Agreed. At the same time, this observer 
^detects a hint of slippage in the data he has examined -- that is, the perceived 
"superiority" of LER may be somewhat lower now than earlier in its existence. 

Of interest, too, in the overall perspective, is the cost-benefit posture of 
LER. By making a few gross assumptions, at least some statements can be made. 
Assume that $600,000 represented the total cost of LER ($400,000 djrectly from 
federal grants and $200,000 from other federal and state sources). Assume also 
that 75 percent of these funds, or $450,000 went for direct training of fellows. 
Finally, assume that 30 fellows had been graduated from LER -- the 25 completers, 
plus 2 near-completers, plus a group in the pipeline equivalent to over three 
FTE graduates. 

This places the cost of training at $15,000 per graduate, a seemingly "fair" 
estimate. Present evidence suggests that nearly all graduates will remain in the 
field; let's assume that 80 percent do and that each works 30 additional years. 
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This amortizes the cost to about $625 per graduate — year of research > 
teaching, and service in tne field [$450,000 f (30 graduates X 80 percent X 
39 years}]. A bargain, you contend — a bargain, I agree. 

^* Strengths , - Weaknesses , and Probabl e Future of LER 

LER continues to maintain its three main strengths: (1) key professional 
staff, (2) bright. motivated students, and (3) provision for considerable inter- 
action between staff and students. These strengths of the Lab are real — not to 
be taken lightly and not to be taken for granted. 

There are, however, here and there, indications that the vitality of LER in 
1973 is somewhat below that in 1969 esprit possiblv not quite so high (both 
student and staff ). internal communication not quite so good, core curriculum and 
experiences appearing not quite so relevant, etc. It is difficult to estimate the 
effect on LER of the changing job market (essentially a tight construction of the 
professorial opportunities) end reduced funding for research training and,t 
seemingly, for research itself. Lack of- student support funds concerns one LER 
staff member considerably — he believes such a shortage has and will increas- 
ingly manifest itself in the form of fewer applications to LER, less qualified 
trainees, self-selection to LER if having access to own personal support funds, 
etc. 

The 1973-1974 academic year is a pivotal one for LER. Dr. Glass is on 
creative leave and reflecting on his appropriate university-LER role. Dr. 
Worthen has jyst departed from the Lab. Federal support of trainees has termin- 
ated. Though all nine current trainees are financially supported via a patch- 
work combination of i/iiversity fellowships, teaching assistantships, and working 
internships, certainty of their funding in 1974-75 is far from assured. The job 
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market for which LLR was created i? currently assuminq a verv low profile. A \ 

1 

new physical location for LER is daily becoming less a ruinor and nrore a source \^ 
of uncertainty (as to potential effect on LER). 

Tiiis evaluator is of the opinion that LER as it was conceived has peaked, 
and irhat substantial and important progress was made in terms of attaining the 
•^Djectives set forth in 1965. Good show -- really first rate. 

•Tnis evaluator is also of the opinion that the long range future viability 
and vitality of LER is linked to a careful reanalysi^ now of the total program 
^-4iiut-the apparent national need, and reconceptual ization to include new or modified 
objectives as appropriatti . Possibly research in substantive content fields should 
. De launched. Possibly student input should be reduced to better match resources 
• available (both financial for student support and faculty time resources for 
•interactions with students). Possibly consulting off-the-street should be 
abandonea or modified in the interest of better training. Possibly basic features 
of tne nresent LlR model should be abandoned or modified. Possibly all the 
, possi Di 1 1 tier I've mentioned above should be reversed. 

Nevertheless, the truly major accompl Isbments of LER over the past seven 
years stand they are a matter of record. It is possible that the next seven 
years miqnt be as productive and filled wi :h accomplishment as the last seven 
even wi tn no sign'^ncant changes in the LER training model. I would, however, 
be surprised if that occurred. With the prior excellent posture of LER in 
regard to considered, -<=»flective response to evaluative data, I am comfortable 
in tne belief that a formal reanalysis of need, objectives, and program would 
result in a stronger LER over the long haul (even if, by sor.e quirk, the 
decisions ultimately reached left the present LER training model' virtually 
iniact). 

\ 

o 

ERIC 



APPENDIX A 



ERIC 



=3 



to 

O 



CD* 



c 

o 

U 

o 



c u 
o 

4^ •#•<» 

: IS- 
t o 
} o 
o 

I ^ E 

: ii 

^ r- a. 

* > o 

I ^ 

c c 

O 

c 

E X2 
3 O 
C 



2C -O 
UJ c 
CO (O 

O 



c: 
Ol 



ERLC 



C 
OJ 

§ 



<TJ O 
r- C 

u 

O 

c 

O Q} 

a; c 
x: 



C 



>> - 

C C 

O (U 



O 

I. O I— 

35 q; 3c^ 
c: 

<0 QJ cu 

0^ <t» c 
o 

01 (A 
CL U 



r5 
o 
>> 

o 



Oi 



> 



its 

c 

o 



c: 
o 



O 



o 
>> 

4^ U 



TJ at 
u > 



> O 



O «J c 




•r- C O 




^ u 


u ^ 








> 




C -r- 
















cn 




c a» 




•r- JC 




3: 




O 


:k (ts c/> 


O 


O 




r- OJ 3 


o 


r- :x O 










OJ VH 




x: o 


OJ • o 


4^ 



^ c 

a;i 
m > 

OJ *r- 



s 



>> 

01 



C 

•s 

o 
u 
u 

01 



* O 
L. 

• 

TJ 
• • I 

VI t/> >s 
U E 

C K 3 

E ro 

(u a> ^ 
> 

•r- * • 

U C d 
ro CU 
(/) x: 4-» 

0^ I— 
C SL. 3 

o E c 
s. o o 

CQ O O 



<M CO 



o 



E 

I— to 
o;i 
i/1 (A 



-M en E CL 
Q> O 

o; J- u 

tn 3 •TSi 

C I T3 in 

O 0^ <T9 t- 

•I- OJ 

+j o>4-> x: 

OS I ro 4-> 

3<4- a: JC o 



> I c • 

0> C >» <XI ^ 

c x: c o 
incoi-^*r^4^303g 

^ ^ ro 4-> 0) 
OJ 3 Oi ^ <ti in 
«n x: u L. 

C 4-> 3 3 
r- O O C O 
O w (O UJ O 



U 01 

3 4-» 

tn <0 



u 

0> 0» 

p 

^ p 
olu. 



(n c. 

c o 
o 

4-* O 

u 

<0 4-» 

$L. ro 

Oi u 

4-> O 

•f- ro 

3s ^ 

o o 

^ u 



in tn 
u o 

p <J 

OL U 

<v 

Oi c 

4-> 

«o 



f 

• Oi 

\c c c: 
-o 

(O 

Oi cn <o 
a o 

1^ 0IJ3 

tn o *»n <TJ 

<U O r- 

4-> T> a. o 

•r- 3. O 
> 4-» T? 

tn Oi d 



o 

4^ JZ 



Of-* .r- r- 

Oi c ^ 

4^ (II 

ro I ro I 

L. 4^ C 4? 3: 

01 p— Of-* o 

tn 3 3 ^ 

in u c u r- 

<0 •r- ro Qi 



0 4J> <0 

Oi 4i4 O 

01 C C 4^ 
Q. Oi*r- C 

t— CL I Oi 
r- OiM-'O 
3 X>r— C 
jD C Oi Oi 



•r- Oi 

^ u 

tn "O 
>l 

f— 4^ 
01 r-~ 
^ 3 

o 

01 t4-. 

x: 

4-» 4-^ 
O 3 

o 

JC JC 



3: 3: 



CO 



<J| o 



Oi 

> 
O 
•O 

<o 

Oi 

JC 



O 

u 

tn 

tn 
<o 

XT 



Oi 

x: 



Oi 

4-> 
'O 

I. 

Oi 
tn* 
so 
Oi 



CM 



tn 
$^ 
to 



tn 
Oi 

>> 

en oc 

UJ 
CO ^ 
Oi 

>* >> 

cs: UJ 

UJ C7> ^ 

^ c: 

E 3 

a >> 





4-» 


3 




3 


u 


•o 






Oi 










4-> 




4-* 










O 








u 


■a 




•o 




Oi 


Oi 


Oi 


4-» 


M 




rsi 




•f— 








tn 


C9 


tn 


'O 


to 


U 


<o 


Oi 


JC 


•r- 




hi 


CL 








E 


O. 


E 




Oi 


Q. 


Oi 


id 




ro 






Oi 




01 




-o 


4J 


> 


E 


c 


O 


o 


Oi 


3 






















o 


1 













tn 
Qi 



JC c 

U Oi 

ro Si. 

Oi 3 

cn u 





Oi 


fO 






3 


o 




•r- 




* % 


c 


•w 








3 


Oi 




tn 


1 1 1 


<o 




Oi 


*#- 




O 










• 


o 




4-» 




ro 


• 


U 


4-> 


O 


cx 


X2 


Oi 


CO 










c 




to 


c; 






4^ 








%^ 


O 


Oi 




at 


• 




-J* 








E 




to 


•o 




c 




ro 








Oi 








XI 








tn 




tn 




o 


o 


C3l 








in 


re 


(O 




c 


ro 


o 


C 


o 




tn 


o 






in 


4-> 


ro 


tn 




Oi 


c 


3 




cr 


3 








01 


Oi 


tL 






Oi 


Oi 




tn 


t 


ro 




OI 


O 1 




u a. 







i- 




3 


6 




Oi 


ro 








o 


Oi 




tn 


o 


ro O 


Oi 








a. 


o 







u 
c: 



o 



en 



CO 



to 

O 

O 
< 

2C 



CO 

or 



a* 



01 

*/> 
c 
o 
a. 
*/> 
a> 

u 
o 



3 

U 
CO 



T3i 



C 



0» 



c 

8 



01 

cn 
> 



cn 
> 



IT) to in if> 

<^ «v> <v> 
CSJ CSJ CSJ CVJ 



CD m in Lr> 
^ ^ ^ ^ 
CO CO fo ro 
c\j eg CO eg 



in in in in 
^ ^ ^ «^ 

CO CO CO CO 
CM CM CVJ oa 



01 






o; 








01 






> 


> 


<o 












>> 


U 


U 






•r* 








c 


4^ 




o 


l/> 






OJ 


Q> 















ERIC 



I. 
QJ 

Oi 
C 

o 



0^ 

cr 



K 
0^ 

c 

OJ 



II p— 



t u 
•o p 

3 C 

O O 
X) 

eO 0^ 



OJs-^ 

01 

(/> 

O </l • 



01 



0^ 3C 

* • * 
0> >- 
r— UJ 

CU 0^ 
0> > 

a. m 



3 (A u 

u o» <ci 

•I- 3 

cr 

01 

^ S 
c m 

(A O U 

3: v> 

<A Gl 



c: 
c 
to 

0» <t3 
O) u 

cnr- 

OJ Q. 
X3 (X 

c: 

(O o 
0> 

8^ 

0 u 

•I— ^ 

01 o • 

(d I— 

•r- U> U 
3 fd fl3 
CD 3; S- 



3r o 



a; 



t/l • — 
3 

26 u 
(d 
01 ^ 

u 

C 01 
3 U 

cn.#- 
c 'q. 

O Q. 

(O 

0> 

HI ^ 
(A r— 

•XT C 
O O 



u> 



u 
o; 

CO 



o; 

s 

cy> 

lA 

c 
o 

iA 



CO in in LO 
<ct^ «t «;r «r 
CO CO CO ro 

CM CM CM CM 



0> 
0> 

o> 

I 

«A 



o 

0> 



fO JZ 

C -M 
O 

•r- 3S 
lA 

lA C 

01 O 

O 4J» 

1. U 
O. <0 

u 

f~- OJI 

o; C 
-a 



cn o 
x> 

0^ x: 
c o> 

<0 O 

x: 



o; 

JZ 

o 



01 

IA 



cn CO 

C 01' 

e b 

•M 01 

*A X? 

CO 
CM 



0> 
iA 



in 

CO 
CM 



CO 
CM 



o; 



cn 

§ 

(A 



0^ 
IA 
Id 

2! 

U 

c 



3 

o o; 

t0 eo 
3 

0) -o 
o 

e CD 

-o 

O C 

SP O -M 

^ 32 
o (O E 

XT U 01 
CA 3 U 

>SLLI 3 

•M cr 
u 

«d I— r- 
^ O O 
O O 

x: x: 
UJ a a 
ir %/) 



o 



01 



c 

01 

% 

o> xj 

X2 

3 
(A O 

4J» x: 

U tA 

^ >^ 
cr 4*> 

Or- 
U 3 
U 
r- (O 
as M- 

O X? 

o c 

CO 10 



00 



0» 
lA 

«o 
o^ 

u 
c 



3 

o 

x: 

(A 
<A 

o 



cn 
c: 
O 

«o 



<o 

c 
o 
u 



o 

CO 



8 



CVJ 





0^ 












4^ 


























o 






















> 





CO 



C 
01 



o 
>> 

&^ 
>» 
jr 

AC 

u 
(/I 



• • 

3: 
o 



3: 



c 

0^ 



> 



o 

c 



01 
13 





wO U*> LD UD lf> 










in 




^ ^ ^ 












CO 


CO CO CO oo ro 


CO 


CO 


CO 


to 


CO 


C\J 


CVf CM CM CM C\J 


CM 


CM 


CM 


CM 


CM 























c 






r— 


0; 


CO 


1 


fO 








01 






1 




O 






















0> 






i 


gr 




as: 










wD 








VO 







O 

O 

c: u 
o 



01 



U3 J3 
O 



x: 



i 



OP 

-a 
ai 
> 

2! 



0> t/> (A (/) 

c 4^ E S 
»f-* C 01 OJ 

Q.»r- Ol CL 
3 

Q err-- 

(/> q; <G c 
c o> 
o 
01 *»- 
l/> u 
iA 
0> 01 
01 ^ 
-O O 

w 



o 



> 



3: 

O 



01 
LU 



4-> 



U 
0> 



Ul 0^ 
iA CO 
OJ 01 

O -o 

o 
o 



> 

0> 

(tS O 
E >i 



o 
o 



c ^ c 

U OJ • 

a> o 

a> o a> o 

c $^ c u 

u x: 



o 

in 
u 

O 



o o 
^ o 

c 



O I— 



I— c 



O) 

0 • JC 

o; 

0> 

J= 3 O 

o> o> o. 

01 U G7> ^ (O 

ITS t. OJ 0>4-> 

$-» CXi— C C 

^ 01 

OJ Of o jm£ «^ 

</l UJ C Jg I— 

OJ 

c O • 



&. 
o 
o 
o 

x: u 

O £ 
I— OJ 

OJ 

u 

CO <0 
(/I 

a> 

c c: 



<o 
x: 

x: 

■M 
its 



c 

OJ 



o 
c: 



OJ 

•i 

OJ 

- e ' 

4-> OJ 



O 3 > 
O U V- 
^ • fO OJ 

C -C <0 'I- 

fc- c cn^oj 
•p- v> c: ^ 
3 OJ -I- <o 
cr4-J > 4-> 
OJ c <o C 
ci^ ac '1- 



4 



OJ 

OJ 4-» 
$^ V- r- 
.<0 ^ 3 
OJ ^ 
QJ C OJ 
1. OJ 4^ 
CLX> (O 
I 1. 
4M 0> 
O C 
C 3 

lO WO U> 

'Si- ^ ^ 

CO CO CO 
CM CM CM 



OJ 

u 

<o "O 
a. OJ 

OJ -M r- 

(-•MS 

O.— 

I OJ 
»— OJ -M 
I— C <o 

OJ OJ t. 

3: .a cn 



rO XI U 







01 


OJ 














OJ 


OJ 


<o 


<o 




OJ 


OJ 


3 


3 




1. 


L. 


cr 


cr 


0 


cn 


cn 


OJ 


OJ 




(O 


(O 




TJ 




(A 




(O 


<o 


lA 






c 


c 


3 










JQ 




u> 


in 




iO iO 




<!■ 








CO 


CO 


CO 




CO CO 


eg 


CM 


CM 




CM CM 


f— • 










OJ 


QJ 


OJ 


0) 




01 








C 






(O 


<o 


.(O 




CP 


3 


3 


> 


10 


<o 


cr 


cr 


OJ 






01 


OJ 








-0 










to 


(O 







x: 












0 












0 3 


OJ 


01 








0 B 


x: 














OJ 




















OJ c c 












t. <o 


0 












LU 








3P C 3 


c • 






• • 




cn tn 


0 (A 












•r- C 


OJ 








0 tn <o 




x: 








•p- OJ 


(0 










x: XJ 














<o <o 










0 OJ 




0 










01 0 








01 


4-> >> 


U 01 


(/I 








u E E 


CL OL 


OJ 




u 




OJ OJ 




•r— 




OJ 




•ox: X? 


1— 01 






c 












01 










• • 


cn 


Ul 




0 E 


•r- 


01 










U 








iA ^ 




<o 


OJ 








(A OJ 




3^ 


CL 


0 


4-» p 


01 E 
















t/> 


x: 


W1 


OJ 3 OJ 


0 E 


<o 


X3 




4-J 


•f- U N OJ 


10 


0 


C 


c 


C 


r-*r- >>X: 


CL 1U 




OJ 


Si. 


OJ 




cn 




cx 


01 






OJ 0 








r— 




x: u 






c 




0 XJ to 0 


1- ct 


a. 


to 


< 


• 


• 


• 


« 


• 












00 

















ERIC 



fe7 



in 



cn 



or 



CO 



3: 
o 



UJ 

a. 
o 

5 



01 



5:. 



-o 
>> 

cn 
d 
o 



in 

CO 



• >> 

CP XT» 



V. 

O 3 
to O 



o 
c 



to 
Of 

on 

e 

CL 



O <l> 
I- -D 



0) 



e 





i 




c 


0> 










4-> JC 


c to "O 

£0 01 










CI 

U o» 










cu 




in r— 










c/> U 












«^ s 


i 












*r» O O 








<t> 




4-* r- O 












«0 r- 








>k 


U 0^ 










o; 












0^ Ou 




















I*— ^ 


o 




3 Ul 






Qd 
Ul 


tr 




Ou— 1 










Oi 




01 




t 






to 






c 






V. o; 


4-> 




c 






0^ 


•r- 




cr 


o 


Ol*^ 


Ol OJ 


h- 


L J 






^ T? 


CO 3: 










4-* ^ 






1 


c 




•r- ^ 


^31 




C= 


o 


o 


1— u 


>sO ' 




O 




x: 


O 


c >^ 




c 




u 






>^ 






m 


• 


01 


4^ 








cn 3 


1— 






3 


01 


• o 


o 


3 




•o 


x: 


a> as 


x: 


O 




UJ 








to 


OJ 






sz 


o; 


u- 






JC 




^^'^ 


cc 




o 














U JC 


to 


UJ 


in 


'o 




o; s 


a> 
•o «- 




01 


o 


trt 


o c 


c: a> 


sz 




xz 


a» 


Q- ^ 


10 


4^ 




u 








•r- 




to 






(/) o. >> 


at 










01 






c 


w 


o u 


•— c <o 


c 






0> 


•r-j4^ 


4^ •r- W 


o 


cn 




to 




•i- </) 








O 




4-» l» 01 


■M 


c 


u 






O O 




o 


o 




4-J i/> UJ 


4^^ 0> 






I/I 






«A c 


o 






Ul c: 


•t- o 






£0 


0^ 


•o o 


^ 4-» 


r— 0> irt 


<0 


U 




O^-i- 










o 




01 10 


0$ 01 




XI 




c to 


tO 3 CO 


to — a> 


o 


UJ 




u 


«J r— U 

OJ <o <o 


<a Q. o 
01 E (O 




* 


* 


UJ *o 


> OJ 


5 Q. 


• 






^ UJ 


O. 01 >> 


D. U CA 





to 

as 

o 



0^ 
Ul 



o 



o 
x> 



o 



CO 
4-> 

c 

01 
T? 
3 
4^ 
t/t 

e 
o 



C 3 

o -o 



1- O 
O 

«j o 
o 

O U 



o; 



c e 
o o 

•f- Jd 
a: 

«- 

o 

jO 

f- 03 

r— a. 



01 

lA 

cn 



01- 



FRir 



O r- 



CSJ 



8 



(A 



i 



to 
o 



LlI 

o 
I— 



. St 

o 
in 

or 



c 




o 
















> 




> l- 




€U 




4^ CA 




U 








a» 








C 4^ 




flS ••-> 








cx E 




•r- O 




JC U 




to 












•i| 












01 








to 








o o 








tfi f— 






C 


01 c 


O 


o 




o 


4-» 






QL ftl 


PN» 


c 






C 


C * 


<T> 




CD 




U 




o 
















o 




O -f- 










c 


4-> O 






x: 










c 




01 o 




iA -r- 




(0 4m» 


• 




u 









0^ 

> 

• 

01 O 
0>«<*^ 

oc 

UJ * u 

=5 lA * 

O tn 4^ 

O -M 

O 

CI. o 
01 

"o* o § 

•r- "r— 
-C ^- 
^ 01 
U 

O 01 0> 
fO (/> tr» 

C 

LO LO O 
C U 
4-> O 
W> 'r- 4-* 

f0 



0) 

-o 
o 

c 

01 

tA 
ro 
0> 



tA JZ 
U 

0> O 



^ Cvj «f to 

• ^ ^ ^ ^ ^ ^ 

a> c c c c c c 

S S & & S 



to 
u 



^ O 

^ x> 

(n 0^ 

E O; o; 

0^ oji cn^ 

4^ ^ x: 
a. o> u 

f- CLXJ 3 

O O 
O O OJ o 

u> in m ur> 

^ iCj" ^ ^ 

CO CO CO 
CM CO CJ CM 



en 

CM 



a. o» L. 



4^ x: f— T> E 

(/> (Q 01 O 

fO U C 
E *r* 0> 01 

C 4->f- 0> 

<o x: 0) (A ^ 

U U 0) 

cn E 0^ 01 

O x: >>r- 
r— O 4->^ -M 

o o i: 

-O 4^ O O 
O O O r-* 

4-» O 
0) o 
E -M 

sz 
u 
V. 

03 
0> 
U) 
0> 



E 

n 
o 

•r- • • 

5u iA 

o 3: 
0) 

s- o 
o 

0> 
C 
LiJ f- 
^ (O 
4-> 
OJ c 
x= o 
I— u 



O 01 

^ I— 



OJ 



§ o 

r- O 
OJ U S! 

O E 
O U 

OJ o d 



c 

c: m 

c 
OJ o 
> u 

E 

0 3 
OJ 

C3l 3 
lA O 
J- 

OJ I- 

a. 

=9 

4^ U 
C 

OJ 

cA a: 

01 LU 

w ^ 

OJ 

i. x: 
o 

cnx: 
c: 4-J 



CM 



Oi OJ 
OJ 

C 3 

o o 



CM 



CO 
JZ 

a. » 
E c 
OJ o 

01 4-* 
r— It) 
4-> 3 
4-> •— 

> . 
OJ 

8 C 
4J O 

CO 
CM 



xr g 

OJ 4-* 

x: 3 
3 «o 



4J OJ 

cA O 

•r— O C 

XJ 4J O 

o 

O 
T> 
O 



c: 
o 

•r~ 

'IS 

> 

O 

O 

m I— 

•r- O 
(A -O 
CO O 

x: x: 

<X4-> 
E OJ 
OJ E 



- 



I 

iA 

E 



3 



to 



0} ^ 
U t/> w 

o 3: *f- 
"5 3 



u 
c 

t 

01 



u 
o 

U 

0^ ^ 

o: 1^ c 
UJ eg 
^ o o: 



00 
CM 



f 



p. 
o 

I— 
o 



:3 
o 
u 

o 
c 

01 

> 



0^ 
01 

♦J 

c 
o 
u 



i 



u = 

0 s 
>S 

O 

01 

&. 

OK 
fO o> o 

- O 
>^ 

01 E O 
C O 
O I. (/) 

4^ in 

c S (d 

0) 

in I — 

o; a> OJ^ 

w i c 

3 31 O 
O r— 
>» O 
O C 

cr O 

4-^ Of eO 
II ^ Ou 

0^ U 

O 4^ O 

^ c o> 

C -r- C 
O 

</) «- <- 

^ c a> 



U 

u 
o 





6 




o 
















o 






• 




1 


o 




o 






E 
















0> 




to 












4-> 


• 




u 


• 








U> 




3 l/> 


o 




c 






2? 


C 


1. 


X3 


0> 












to 


4^ 


x; 












C 










4^ 


O • 




8 




O 




O 


i. 






Ml 


o 




O 


<XS 




rO 




W1 






x: 


CX 




C 


O 


01 


o 




a 


0) 




o 










1/1 




> 1/1 
















•r^ (/) 


*5i 










>» 


c 


W 0^ 


<s 


q: 




01 




4-> 


o 


CL %fi 






Ol 




•r" 


•r* 


m ^ 




c 


(IS 






U 






<o 


E 

(A 






0> 


O 


C f— 


c 


c 








o> 


3 




•r- 




<o 








*o 


C 














0) 


4-» o 






C 










C -1- 


c 


c 










OJ 


















E « 


i 




C7> >> 


D1 






<d 


OLSC 








c: 








O 


o 


O 












r-» * 






x: t» 


x: 








0) • 




0> 




u 




L. 


<o 


> 


> 


> 


<o OJ 


<ts 




O 




OJt • 


Oi 


01 


01 > 


0^ 








'X^ 0J» 


*o 






•M 




U 


o 






c 












T? 


x> 


•O 3 


•o 








C p 


c 


c 




c 




•r- 






to 






<o 




T5 


o 








o 








4-^ 










x: 


x: 






u 


u 


a 01 


o 




o 


c 




1. 


1. 


W.s01 


1. 




(. 




<o • 


<o 


<o 




Its 


01 


<ti 






01 


0) 


Oi 1^' 


01 


> 


01 


1. 


01 J3» »— I 


C/) 


1/1 




tA 






O 




Re 




S8 


Si 


un 


Re 


Co 



t/1 



VI 
4-» 

O 



Ml C >> 
<0 O 4-> 
X> L> 

lA 

0» C31 
U C 0) 
C -r- > 
«9 

I— <0 C 
0) 

01 tA 
t- 0^ 

x: 

u o 
* x: 
4-> <t» 3: * 

CO) ^ 

o; <A 0> 

O; C 01 

C O (A 
0» ^ 

CL-O 4-> jO 

01 01 O 

*o C I 
c: o I 

-I- r-* 4-> 

Q. 3 C 
C g ^ O 
^ OF 

C 4-3 ro 

O tA Ni 

01 0> 

^ CA I. C 

U 01 <d 

L» * 4-i C71 

a 01 *r» O 
CA • 



E 

0> tA 

•r- O 
4-> 

C U 
•I- 01 

lA •r-- 
C O 

-M Ul 

3 a> 
u xz 

o 

Oi oI 

XT ^01 



I 3 
I O 

tA 4-» 

Oi « 

^ -E 
o ^ 



tA 
Oi 

> 
c 

3 

<d 



t 

Oi 

X) o 
c I— 
10 4-> I— 

Oi 

M- >> Oi 

Oi ^JZ 

^ E 4-» 

o. 

|A t • 

• E 
^ Oi 
3 I— x: 

g 01 ^ 

>> Oi o> 
C r C 
4-> O CO 
<XJ CXr ^ 

x: E w 
4-» o o o 

0 4-> $ 

4u» ^ 4^ 
!U a cA o 
«S fc- 01 ^ 
Ol fO X: 

01 C71 Oi 

4-^ tA O 

= c 
Oi 



O -H 



u 

Oi 

O O H- 



CSJ 



ERIC 



3 C 



CM 




00 



CM 




a: 

8 



5/> 



o 



IS 

a. 



c: cn 
C3 c 

o , 

I— r— 
3 r— 
U O 

f JC 
I 4^ 

I- ^ 
o <u 
o. 

a: > 



C x: o 

4-» CL 



01 



3 
O 

O 



C 



•a 

Oi 

u 
c 

OJ 



c 

0^ 

K 

0^ 




CM 



4-* 

o 



in 
c 



(Q IS a> 
^ O Q. 



CVJ 
O 



Q> 

c 

01 

O K 
I— 01 

CO 
CSJ 



CO 



3 




0^ 


+-> 




Oi x: 


in 














* 


3 0^ 






















3 






U 


3 




<o 


o o 




u. 








<t> >> • 




c: 


4-> C 


O 


o 


Or- O 






r- 3 






Q U -M 








u 


u 




3 


3 


Vh- 3 


T? 




o -o "O 




UJ 


C 01 












4^ 


O 


O 


VI O 










»— • 




O 


o 


01 o; O 


o 


o 


> 13 O 


XZ 


XZ 


•r- 3 ^ 


u 


a 










♦ 


• 


• 




J3 





c 

s. 

01 



o 



3 cn 
o c 

C 
3 

0> 
UU 3 



4/> 

c 

0> 
3 



o; 



,01 

E 3 3 
4^ 0» ju 

o; 

Ol 3 O 



O " 



O 



y <0 
3 

t/1 OA X» 
C OJ 
O trt 

0 U >Ti ^ 

c: * c 

01 s <o 
o 01 I cn 
u (t> c 

0* ^i* 
O. 4 -M 
0> XJ I— 

oj> x: 3 
x: u> 

4-» C C 

C o 

0> E 

<M E 

<tS (O 

E OJ _ 

•J— 4-> 0> 
4^ O O 
c/) C 
UJ w» Q. 



cr 
OJ 

3 
4^ 
l/> 

OJ 
4J 

to 

3 

. ■ 
3 

u c 

CO o 
4^ 

S ^ 
o u 

1- 3 

4-» -O 

<0 OJ 

U t 

3 cr 

Tl O 

OJ c 

x: x: 

4-> 4-> 



3: 3: 



cn cn 
c c 



3 3 
C C 



01 

c c 

4<> Q 

. 3 

o &. 

I. 3 
O 

OJ 

4^ I. 

3 O 

cm- 



• o e 

4-» (A C 

<0 OJ 

<o x: 

C 

d <0 •r' 

O ^ 3 

•p- (O 

4-J 4-» C 

U 4-> 

3 (A 

OJ C *o 

C C lA 

O C <o 

C 3*^ 
i- 

x: <A 

4-> Si. 4-> 

•r- O C 

cn C r- 

•r- E 

r— <0 UJ 

3 U _|f 

tA cn 

C O 3 

O L. 0|4i> 
O (A 



^A 
OJ 



M M M ^ 



XI f— 
OJ *r- 

o» C <o 

L. 

O (O 01 

a u > 

0.4-» o 



■O CM 

<0 3 r- 
O 

C OJ OJ r- 

3 i» 3 4-> C <0 

0 O O 4-> U 

>^ OJ CO OJ 
(O X> i- > 

OJ 4-> 4i> O 
>, O <o 
r— C JC 
n- OJ 4-> 
I. T- 

E OJ 01 
OJ 
K ^ 

01 OJ 

3 

•O 4-J O 
OJ U >> 
(A OJ 
3 *r-> O 

O o -o 
o w 

c x: cA 

OJ U (O 

OJ 3t 
JO «0 

OJ 4-> 
1^ tA •r* 

OJ O^ 

to 

OJ 

r- e 

C CJ ^ 

•f- iA 4-^ 

C I 

•r- OJ 

CO cnoi 
5- J-x: 

4J CO 4^ 
f— CO 

UJ OVw' ** 

^ c c 

CA OJ 
OJ 3 OJ OJ 

x: c OJ jQ 

4-> C 

4-> •I* OJ 
X> C CO > 

<o o i- <o 
X o 4-» x: 



O r- 

c: (Q 
in 



CM 



(O 4^ 4> 

CO c: 

O OJ OJ 
4^ t. 4-» 
OIK 
OJ 



U 01 c 
O C 

Q. f=: c^- 



o 
o 



iA 

u 



U C 4-> 

_ l-i- CO OJ 
CVO t. 4J 

' c 

o 

o 

0> 

Q.4J 4J 
<A U iA 

CO •r* 4«> 
4-> CO 
C *A 44 

3 t3 

O -O 

OJ CA 0> 
C «l O 

O •r' *N-^r— 

^ CO O 
U CO T3 

OJ >J<0 44 



CO 
OJ 



OJ 
OJ 



X OJ 
OJ 4* 
<0 

^ 3 

„ OJ 

3: 

(O 

c 



CO 



OJ 
E 
c 

OJxr 

4J u 

O CO 
U OJ 
(A 



OJ 




4J 




CO 




-a I. 




OJ 




V. 3 




(O o 




CL >> 




E 




o -o 




U r- 
3 








OJ % 
















x: 








CO ♦* 




C iA 




O OJ 




3 




44 cn 




<o «o 




CX OJ 




3 r— 




U 




U O 




o u 




44 CC 




C UJ 




OJ -J 




iA 1 




OJ C 




u o 




C3L C C*- 






S- 


r— 


3 3 


01 


o o 


tA 


>> >> ^ 




3 


c o 


O 


M 44 


>> 



o 

OJ 

L. C CO 

OJ-r- S- 

CO OJ 

C L. > 

•r* 44 O 

CO 
CM 



O X? r- 
•r- OJ 

U C CO 

OJ— I. 

QL <0 OJ 

3 I. > 

(A 44 O 



UJ o 

ERIC 



CM 

CO 



CO 
CO 



CO 



7/ 



8 





1 






\n 












u 






U C f- 






:3 (t) Qi 






</> JC 0} 




4^ C 






(O <0 01 






o;i 'M 






O I- K 


P ^ o 


O 


4ui 0> 01 




2= 



in 



o 



1^ 

or 



"O o 

1/1 o 



^ 0^ 

a> A 

tA EH 
c o 

o o 



<0 

> <a u 

•r- 0> 

1— a> 
c: c 

O (t> c 
5s lU 

4-» fXS 

0 L. 
e 4J 

C 0>i— 

01 o a» 

<M U CO 

K £X 
0> 

«U -I- 

x: c 
3C 

O O 



O <0 

>> o ^ 

= EC 

3 1- CTW^ 



O 
CX 



in 
OJ 

u 

U Oi 
= 3 

5 ^ 



* u 

— 4-> 
L. 

OJ OJ 
> r- 

o >> 

C t/> 

OJ 
C 

0 ^ •t*' "r^ 
4^ r— 

C = - 
i/i OJ I— •» 

OJ -M p 
4-» (AW 

3 OJ 
r- O 
3 >^ U 
U 3 C 

01 CA O 
a. (tS S •r-* 













OJ LU 
lA ^ 1 










cv 




3 >» Q. 




en 




O Jd 3x. 
>» O 




c: 


4-> 




t 


c 


fO 


cw o 


CA 






OJ 


CA 


<a 


1— 


TJ OJ 






O 


•f- C 




4^ 




tA 4- 








c o -o 


3 <a 01 


L. 




O 3 


(A f 01 


OJ 




C-> 1— 


4J ^- 








lA 


4-> 




2 x: c 








O 4^ 


OJ Jl O 


3 




c 


^ c 














« OJ 
















3 4^ c: 








O OJ 








3K CA 




TJ 




01 




OJ 



01 



OJ 

sz 



OJ 
OJ 

^- 
o 



C 4-> 
OJ C 

K O 
OJ CL 

<M CA 

x: j= 
3: ^ 

o 









OJ 




x: 




4^ 


CO 








CM 


OJ ^ 




OJ -a 


rr 


^- OJ 




OJ 


4-> 


4-J cz 


(Q 


CA 


01 






§ 1a 




OJ 




3 x: 


(O OJ 




4J 




O K 




4J 41 


x: 






•f- 3 




x: o 




3: 




c - 












JO (A 




OJ <o 








(0 ^ 




c 




OJ <T9 




(A L. 








x: 




4J 




c 



OJ .p- 

1- 1- 
O 4J 

^ L. 
C 01 

J- 4-> 

C 3 
•f* C*M. 



U T3 • 
OJ OJ U 
OL OJ 4-> 
</> C OJ 



01 

0>4ui 
^ OJ 
O X 

4^ OJ 
CO 

c 

.1- ^ 

•r— CO 
U CM 



4-> 

<o 

•M 
O 
C 

in 

OJ 



QJ 

C7>4-> 
C 
CO OJ 
•4-> 
O K 

in 
en 

CVI 

f— o 



4-* 

o 

CO 
CNJ 



^ CU CO 



c 



c 
a: 





4^ 




4-» 




4-> 


lO •* 








OJ 


<U 




CO 




CO 


E c: 




to 




• 






01 






o 




QJ 






«M 1— 


w 


s- 














O 


c«- 




X> O 


♦f- -M 




0>4-> 










C 


OJ C (O 


<o 






c 






4-> L. 


CO 


Q} 


U 


C3f> I- 




CO 


OJ 


c: 




C 




4-» 


3 


c o> 






•M 






<tS OJ 


o 


X 




OJ 




o 


K 


c: 




4-* SZ 




01 


O 


^ 4-» 






OJ 






4-» 






L. 


J- c 






(tS 




O 






CX 


O 












CL CT 
















4-> 




E c: 






01 




• 






OJ 










> 


OJ • 


u 






L. 




cA 3 






<o 

JC 


OJ 
CO • 


4-> 

OJ 






&. 




•f- Xf 
















O 






















1 


4-> OA 










>s 










Hi- $^ 








>J>s -M 








4-> 


01 








<a 


•r- 










4-J CL 






4J 


4-» 








OJ 


C 








<o 


<o O 








c 










JC c- 


JC 4-» 


X3 






OJ 




4-> 






4-> "O 


4-> 


<o 






4-> 










C 


4-> 






x: 




OJ "5 






a> 'oi 


a> o 


C 






4-> 


•r- 
O 


[LU 

Aa 






OJ 


> 

OJ 4^ 


no 








CX 










•r* <0 


u 






OJ 










OJ 


f— O 


u 






OJ 


CA 


OJ 






OJ x: 


OJ 3 


CO 








x: 


JD x: 

CA 






XJ 4^ 


X> T3 
OJ 








3 


4-J 


3 'r- 






3 I- 


3 








O 




O 






o o 


O C 








>>4-> 


>^xa 




















3 














O 




O CXC-. 




O 


O CA 


"o 






"O 




o 


E 
<o 




u 


"O <o 
OJ 


ho 






4J 




4-> 4^ 






4-J 




U 






c 




C 


cn 




C 4-> 


c <o 


CA 






OJ 


QJ 


OJ CA 


o 




OJ 


OJ 








4-> 




4^ 4-» 






4^ &. 


4^ u 


OJ 








X C 


CL 




X u 


K -r- 


> 






OJ 




OJ 01 






OJ 


OJ 4-> 


•r- 


















(O 


4-> 






4-> 


QJ 


4^ 3 


cr 




4-> U 


4-> e 


<tl 






<ts 


c 


fO 4-> 


**— 




lO 


CO OJ 


£= 






x: 




jc: (A 


c: 




x: lO 


JC r— 


&. 






3t 


ro 








2 QJ 


3C xa 
o 


OJ 
f-> 






o 


c 


O UJ 








o u 












J— ^ 


4J 






1— CX 


fO 







Jj if) 

ERIC 



CO 



00 



cn 
oo 



o 



7Z 



O 



to 



CL 

o 

ex. 



8 



4^ U 

L> O 
C 

4^ 0) 

I/) CL 

•I- 3 



o 








o 




•o x: 






CVi 






S. 




E JZ 


>s 


o u 




u u 


4^ U 




U O 




c 








^ q; 


cn u 




o 


c 






CL • 




Of 




cr> . 












c 








« g. 












01 

UJ S 













JC 



01 



0^ 

^ Cj 

to j 
O ^ 

>> o 

^^ U. (/> 

x: -o 



o 
3e 



0^ o 

<U -C r- 

o -o 



> 

o 



CVJ 



o --^ 

o 



u o 
•f- J- 

(/> CL 

CO 
CM 



U O 
4^ 0^ 



c: 

4-> 



01 



-J >> 

»^ 

4-» 

(1> C 

4^ a> 

O 

O 

3 Ql 
O 

^ o 
I— 

Oi sz 

> ^ 
o o 



tu > 

^ o 

0) CL 

x: E 



c u 

> Q} 
•f- > 

0) in 

CL O 
>> 

O 4-» 
3s «> 

4^ 4^ 

m 

o 

QJ 4J 

0 0> 

01 on 

>> 

0* c 



u 

> 

c * 



-C > 
U 

t^4^ 

CO c: 
01 OJ 
to ^ 

or 3 
o 

CO <o| 

c 

o a 

4^ 3 

L> U 
3 C 

- ^ 

0 <Q 

ire 

o 

4-> 

<o • 

o 

jQ • 
<0 4J» 

0> 

c: 

«Su 

01 
•4J 

^ m 

E*^ 

01 O 

^-o 

3: 

• QJi 

•r- 

0 i/^ 
4-> 00 

O 

01 a. 
w 

c 
o 







«0 E 










Oi 




U3 








11 


a. 


U 




y% 




OJ 


UJ 


-0 






B 


QJ 




tA 


u x: 


<o 


0)4^ 


a> 








ol 


D. 0 



c: 
O 
o 
4^ %n 
in 

0) to 
cr 

c 

X> J- 
OJ 3 
4J 4J 

^ 2! 

CL 

E oi 

O c/> 

u n) 

Oi 
C r— 



0^ • 

o 

XI 

01 

in u 
o 

OJ r— 
I— o 
CL O 



ERIC 



13 



APPENDIX B 



ERIC 




QUESTION SETS USED WITH LER PRINCIPAL STAFF; FALL, 1973 



A-1 : Over the seven-year history of LER, there have been several drop-outs; Is 
it possible for you to characterize the drop-outs as a group (i.e., did 
they have common characteristics)? 

A-2 : For several of the early years of the Lab, monies were received to develop 
the research methodology program. Comment upon the present program as 
compared to the situation existing before LER was initiated. 

A-3 : Has the effort you have expended in developing and operating the LER model 
been worth it or not? Elaborate. ; 

A-4 : Rank order the importance or weight of the following LER entrance criteria. 
(Give a rank of 1 to the most important, etc.) 

1) GPA ' / 

21 GRE 

3) Math background 

4) Commitment to educational research 

5) Endorsement from known, reliable source 

A-5 : Do the high entrance criterion for LER students have any undesirable 
consequences? 

A-6 : To what extent do you feel the students of the School of Education have 
been influenced by LER? Be specific in detailing examples of this 
influence. 

A-7 : To what extent do you feel the faculty and students in departments other 
than education have been influenced. by LER? Be specific In detailing 
examples of this influence. 

A-8 : Comment on how you would view the following in terms of appropriateness 
for the LER training model: 

1) a large scale research project in which students could be actively 
involved. 

2) increased internships with "applied" agencies, that is, public schools, 
state departments, and the like. 

A-9 : Should LER students be required to do a small research project from 
conceptualization to completion? 

A-lO: How important is it that LER fellows teach while earning the Ph.D.? 

A-ll: Do you view the LER student of today, as compared with r966, as taking 
more or fewer offerings in the various behavioral sciences? 

A-1 2: At one point, internships were reduced from 600 to 400 hours. What do 
you view as the benefits and consequences of this change? 
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A-13: 'Have you noticed a shift in where LER graduates are employed? 

A-14: Assuming that the current trend toward difficulty in finding student 

support funds continues, what effect do you predict this will have on the 
LER program and students'? 

A-15: To what extent do you feel that LER students are handicapped by their lack 
of concentration in a substantive area (i.e., an area other than research 
methodology itself)? \ 

A-16: To what extent are LER s^udei^ts in touch ^with the major problematic areas 

in education today {X^^ integration, aHernative schooling, accountability. 



A-17: In your opinion, and considering similar programs to LER across the country, 
how would you rank the LER 'program in terms of Its contribution to the 
field of educational research? 



A-18: To what extent do you feel that LER graduates have or will become leaders 
in the field of educational research, broadly conceived? (Be specific, 
that is, how many do you feel will attain what level of prominence?) 



A-]9: What do you^perceive as the greatest weakness of LER? The second greatest 



etc.)? 




weakness? 
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B-1 : At Che outset of LER in 1966, it was envisioned that several departments, 
other tlian education^, would be meaningfully involved in the LER student's 
program. To what extent do you feel such inter-drscipl inary liaison has 
been achieved and also comnent on the value of such liaison? 

B-2 : Early in the life of LER, it was anticipatvu i . at. there would be 24 

irtudent^ training at a single time in the Lc If such a number had been 
attained, what do you view as the likely benefits and consequences? 

B-3 : To wh?t extent do you feel the success of the LER program has been depen- 
' uent upon recruitment? Upon high entrance standards? 

B 4 : T6 what extent do you see LER as being vulnerable to the charge of academic 
"elitism"? What positive aspects has such "elitism"? What negative 
aspects? 

/ D-5 : Speculate on how successful LER studct.is would have been after completing 
the Ph.D. if they had completed it in an area other than LER. 

B-6 : To wnat extent do you feel that the internships (G08,'^09, 610, 611) are 
critical for ^he LER program? 

£-1 : To what extent would you estimate LL.C students are involved in the following 
activities via tjieir internships? Estimate for both early Lab (say, 1966- 
63) and late Lab (say. 1971-73). 

1) Consulting with graduate students. 

1) Consul tinq witn faculty. 

3) Running data analysis computer routines. 

B-a : Co.TTTient upon the nature of your personal involvement with your students, 
that is, its duration, its intensity, its merit, etc., particul 'trly as 
compared to involvement with students displayed by other faculty. 

B-9 : Trace the history of the brown sacks. Have they recently been what you"^ 
want them to be? 

3-10: To what extent do you feel the faculty of the School of Education have 
been influenced by LER? Be specific in detailing examples of this 
influence. 

B-ili At tines, the School of Education faculty have expressed concern about the 
accessibility of LER given its location in the Education Annex. Some have 
suqqes-ed that it be moved to Hell ems Annex. What is your reaction to 
this suggestion? 

B-1 2; Do you notice a trend in terms of the number of LER students who leave the 
program .for employment before completing their dissertation? 

B-1 3: 'To what extent do you perceive- maicr ciKinyr"- in the competencies acquired 
by a LER student as he completes the Ph.D.. 
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B-14: Several students in reflection feel they would have gained via more time" 
(i.e., another year) 1n>I.ER. How do you feel? 



8-15: Should LER students be involved in preparation of research proposals for 
funding? 

B-16: To what extent do you feel that it is imoortant for LER students to publish 
fonnal papers before completing the Ph.D.? 

' B-17: To what extent do you feel that the faculty and students of the LER have 
produced research critica"" for the field? 

B-18: What would you view as LER's overall impact on the field of educational 
research?- Estimate what LER's impact is likely to be 10 years from now. 



B-19: What do you perceive as the greatest strength of LER. The second greatest 
strength?. 
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